Oct 30 17:11:39.946155 ip-10-0-139-222 systemd[1]: Starting Kubernetes Kubelet... Oct 30 17:11:40.394600 ip-10-0-139-222 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:11:40.394600 ip-10-0-139-222 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 17:11:40.394600 ip-10-0-139-222 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:11:40.394600 ip-10-0-139-222 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:11:40.400518 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400427 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:11:40.400518 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400514 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:11:40.400518 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400518 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:11:40.400518 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400522 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:11:40.400518 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400525 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:11:40.400720 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400528 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:11:40.400720 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400531 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:11:40.400720 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400534 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:11:40.400720 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400536 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:11:40.400720 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400539 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:11:40.400720 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400541 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:11:40.400720 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400544 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:11:40.400720 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400547 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:11:40.400720 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400549 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:11:40.400720 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400552 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:11:40.400720 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400555 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:11:40.400720 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400557 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:11:40.400720 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400560 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:11:40.400720 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400562 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:11:40.400720 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400566 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:11:40.400720 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400568 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:11:40.400720 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400571 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:11:40.400720 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400573 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:11:40.400720 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400577 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:11:40.400720 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400580 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:11:40.401227 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400582 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:11:40.401227 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400590 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:11:40.401227 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400593 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:11:40.401227 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400596 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:11:40.401227 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400599 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:11:40.401227 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400601 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:11:40.401227 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400604 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:11:40.401227 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400606 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:11:40.401227 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400609 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:11:40.401227 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400611 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:11:40.401227 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400614 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:11:40.401227 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400617 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:11:40.401227 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400620 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:11:40.401227 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400624 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:11:40.401227 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400627 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:11:40.401227 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400629 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:11:40.401227 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400632 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:11:40.401227 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400635 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:11:40.401227 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400637 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:11:40.401227 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400640 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:11:40.401693 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400643 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:11:40.401693 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400645 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:11:40.401693 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400648 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:11:40.401693 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400650 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:11:40.401693 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400653 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:11:40.401693 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400658 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:11:40.401693 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400663 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:11:40.401693 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400666 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:11:40.401693 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400669 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:11:40.401693 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400671 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:11:40.401693 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400674 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:11:40.401693 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400677 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:11:40.401693 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400680 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:11:40.401693 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400682 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:11:40.401693 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400685 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:11:40.401693 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400687 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:11:40.401693 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400690 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:11:40.401693 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400693 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:11:40.401693 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400696 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:11:40.402159 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400698 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:11:40.402159 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400701 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:11:40.402159 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400704 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:11:40.402159 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400706 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:11:40.402159 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400709 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:11:40.402159 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400712 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:11:40.402159 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400715 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:11:40.402159 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400717 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:11:40.402159 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400720 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:11:40.402159 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400723 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:11:40.402159 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400726 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:11:40.402159 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400728 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:11:40.402159 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400733 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:11:40.402159 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400736 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:11:40.402159 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400739 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:11:40.402159 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400742 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:11:40.402159 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400744 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:11:40.402159 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400747 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:11:40.402159 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400750 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:11:40.402632 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400753 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:11:40.402632 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400755 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:11:40.402632 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400758 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:11:40.402632 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400760 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:11:40.402632 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400763 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:11:40.402632 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400766 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:11:40.402632 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400768 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:11:40.402632 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400770 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:11:40.402632 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400773 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:11:40.402632 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400775 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:11:40.402632 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400778 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:11:40.402632 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400781 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:11:40.402632 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400783 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:11:40.402632 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400786 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:11:40.402632 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400790 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:11:40.402632 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400792 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:11:40.402632 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400795 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:11:40.402632 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400798 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:11:40.402632 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400801 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:11:40.403091 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400804 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:11:40.403091 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400806 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:11:40.403091 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400809 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:11:40.403091 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.400811 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:11:40.403091 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401621 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:11:40.403091 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401630 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:11:40.403091 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401632 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:11:40.403091 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401635 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:11:40.403091 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401638 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:11:40.403091 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401641 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:11:40.403091 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401644 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:11:40.403091 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401647 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:11:40.403091 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401649 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:11:40.403091 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401652 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:11:40.403091 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401656 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:11:40.403091 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401659 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:11:40.403091 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401661 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:11:40.403091 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401664 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:11:40.403091 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401667 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:11:40.403091 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401669 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:11:40.403597 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401672 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:11:40.403597 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401675 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:11:40.403597 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401677 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:11:40.403597 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401680 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:11:40.403597 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401682 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:11:40.403597 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401685 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:11:40.403597 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401688 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:11:40.403597 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401690 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:11:40.403597 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401693 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:11:40.403597 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401696 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:11:40.403597 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401700 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:11:40.403597 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401703 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:11:40.403597 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401706 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:11:40.403597 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401708 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:11:40.403597 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401711 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:11:40.403597 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401714 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:11:40.403597 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401716 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:11:40.403597 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401719 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:11:40.403597 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401721 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:11:40.403597 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401724 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:11:40.404098 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401727 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:11:40.404098 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401730 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:11:40.404098 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401734 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:11:40.404098 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401738 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:11:40.404098 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401741 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:11:40.404098 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401744 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:11:40.404098 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401746 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:11:40.404098 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401749 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:11:40.404098 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401752 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:11:40.404098 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401754 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:11:40.404098 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401757 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:11:40.404098 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401760 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:11:40.404098 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401763 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:11:40.404098 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401765 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:11:40.404098 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401768 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:11:40.404098 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401770 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:11:40.404098 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401773 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:11:40.404098 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401775 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:11:40.404098 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401778 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:11:40.404592 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401780 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:11:40.404592 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401783 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:11:40.404592 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401786 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:11:40.404592 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401789 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:11:40.404592 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401792 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:11:40.404592 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401795 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:11:40.404592 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401797 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:11:40.404592 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401800 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:11:40.404592 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401802 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:11:40.404592 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401805 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:11:40.404592 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401807 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:11:40.404592 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401810 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:11:40.404592 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401812 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:11:40.404592 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401815 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:11:40.404592 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401817 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:11:40.404592 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401820 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:11:40.404592 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401822 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:11:40.404592 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401825 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:11:40.404592 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401827 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:11:40.405056 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401830 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:11:40.405056 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401833 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:11:40.405056 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401835 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:11:40.405056 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401838 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:11:40.405056 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401841 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:11:40.405056 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401843 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:11:40.405056 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401846 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:11:40.405056 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401848 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:11:40.405056 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401851 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:11:40.405056 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401853 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:11:40.405056 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401855 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:11:40.405056 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401858 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:11:40.405056 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401876 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:11:40.405056 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401879 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:11:40.405056 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401882 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:11:40.405056 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401884 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:11:40.405056 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401887 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:11:40.405056 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401890 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:11:40.405056 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401893 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:11:40.405500 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401896 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:11:40.405500 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401898 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:11:40.405500 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401901 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:11:40.405500 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401903 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:11:40.405500 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401906 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:11:40.405500 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401908 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:11:40.405500 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401911 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:11:40.405500 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401916 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:11:40.405500 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401920 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:11:40.405500 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401923 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:11:40.405500 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401926 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:11:40.405500 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401929 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:11:40.405500 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.401931 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:11:40.405500 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402414 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:11:40.405500 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402420 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:11:40.405500 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402423 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:11:40.405500 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402426 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:11:40.405500 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402428 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:11:40.405500 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402431 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:11:40.405966 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402434 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:11:40.405966 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402437 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:11:40.405966 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402439 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:11:40.405966 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402442 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:11:40.405966 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402445 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:11:40.405966 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402447 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:11:40.405966 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402450 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:11:40.405966 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402453 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:11:40.405966 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402456 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:11:40.405966 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402458 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:11:40.405966 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402461 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:11:40.405966 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402464 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:11:40.405966 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402467 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:11:40.405966 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402470 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:11:40.405966 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402472 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:11:40.405966 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402475 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:11:40.405966 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402477 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:11:40.405966 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402480 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:11:40.405966 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402483 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:11:40.405966 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402485 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:11:40.406451 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402488 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:11:40.406451 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402490 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:11:40.406451 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402493 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:11:40.406451 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402496 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:11:40.406451 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402499 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:11:40.406451 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402502 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:11:40.406451 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402504 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:11:40.406451 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402507 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:11:40.406451 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402509 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:11:40.406451 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402513 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:11:40.406451 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402515 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:11:40.406451 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402518 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:11:40.406451 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402521 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:11:40.406451 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402523 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:11:40.406451 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402525 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:11:40.406451 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402529 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:11:40.406451 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402533 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:11:40.406451 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402536 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:11:40.406451 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402538 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:11:40.406941 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402541 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:11:40.406941 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402543 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:11:40.406941 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402546 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:11:40.406941 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402549 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:11:40.406941 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402551 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:11:40.406941 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402554 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:11:40.406941 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402557 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:11:40.406941 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402559 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:11:40.406941 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402562 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:11:40.406941 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402564 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:11:40.406941 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402566 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:11:40.406941 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402569 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:11:40.406941 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402572 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:11:40.406941 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402575 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:11:40.406941 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402577 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:11:40.406941 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402580 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:11:40.406941 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402585 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:11:40.406941 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402587 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:11:40.406941 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402590 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:11:40.407386 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402592 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:11:40.407386 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402595 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:11:40.407386 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402597 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:11:40.407386 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402600 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:11:40.407386 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402603 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:11:40.407386 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402606 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:11:40.407386 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402608 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:11:40.407386 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402611 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:11:40.407386 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402613 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:11:40.407386 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402617 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:11:40.407386 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402619 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:11:40.407386 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402622 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:11:40.407386 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402626 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:11:40.407386 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402629 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:11:40.407386 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402631 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:11:40.407386 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402634 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:11:40.407386 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402637 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:11:40.407386 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402639 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:11:40.407386 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402642 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:11:40.407386 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402645 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:11:40.407960 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402649 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:11:40.407960 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402652 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:11:40.407960 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402655 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:11:40.407960 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402658 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:11:40.407960 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402660 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:11:40.407960 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402663 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:11:40.407960 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402665 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:11:40.407960 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402668 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:11:40.407960 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402670 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:11:40.407960 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402674 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:11:40.407960 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402677 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:11:40.407960 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402680 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:11:40.407960 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402682 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:11:40.407960 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402685 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:11:40.407960 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402687 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:11:40.407960 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402690 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:11:40.407960 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402693 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:11:40.407960 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402696 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:11:40.407960 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402699 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:11:40.408404 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402701 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:11:40.408404 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402705 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:11:40.408404 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.402708 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:11:40.408404 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403835 2572 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 17:11:40.408404 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403845 2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 17:11:40.408404 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403852 2572 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 17:11:40.408404 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403856 2572 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 17:11:40.408404 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403872 2572 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 17:11:40.408404 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403876 2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 17:11:40.408404 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403880 2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 17:11:40.408404 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403888 2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 17:11:40.408404 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403891 2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 17:11:40.408404 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403894 2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 17:11:40.408404 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403898 2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 17:11:40.408404 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403901 2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 17:11:40.408404 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403904 2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 17:11:40.408404 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403907 2572 flags.go:64] FLAG: --cgroup-root="" Oct 30 17:11:40.408404 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403910 2572 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 17:11:40.408404 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403913 2572 flags.go:64] FLAG: --client-ca-file="" Oct 30 17:11:40.408404 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403916 2572 flags.go:64] FLAG: --cloud-provider="external" Oct 30 17:11:40.408404 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403919 2572 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 17:11:40.408404 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403924 2572 flags.go:64] FLAG: --cluster-domain="" Oct 30 17:11:40.408404 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403928 2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 17:11:40.408404 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403931 2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 17:11:40.408996 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403934 2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 17:11:40.408996 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403938 2572 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 17:11:40.408996 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403946 2572 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 17:11:40.408996 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403949 2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 17:11:40.408996 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403953 2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 17:11:40.408996 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403956 2572 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 17:11:40.408996 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403959 2572 flags.go:64] FLAG: --contention-profiling="false" Oct 30 17:11:40.408996 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403963 2572 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 17:11:40.408996 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403966 2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 17:11:40.408996 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403969 2572 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 17:11:40.408996 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403972 2572 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 17:11:40.408996 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403976 2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 17:11:40.408996 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403979 2572 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 17:11:40.408996 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403982 2572 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 17:11:40.408996 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403985 2572 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 17:11:40.408996 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403988 2572 flags.go:64] FLAG: --enable-server="true" Oct 30 17:11:40.408996 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403991 2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 17:11:40.408996 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403996 2572 flags.go:64] FLAG: --event-burst="100" Oct 30 17:11:40.408996 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.403999 2572 flags.go:64] FLAG: --event-qps="50" Oct 30 17:11:40.408996 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404002 2572 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 17:11:40.408996 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404005 2572 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 17:11:40.408996 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404008 2572 flags.go:64] FLAG: --eviction-hard="" Oct 30 17:11:40.408996 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404012 2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 17:11:40.408996 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404015 2572 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 17:11:40.409591 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404018 2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 17:11:40.409591 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404021 2572 flags.go:64] FLAG: --eviction-soft="" Oct 30 17:11:40.409591 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404024 2572 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 17:11:40.409591 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404027 2572 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 17:11:40.409591 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404030 2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 17:11:40.409591 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404033 2572 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 17:11:40.409591 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404038 2572 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 17:11:40.409591 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404040 2572 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 17:11:40.409591 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404043 2572 flags.go:64] FLAG: --feature-gates="" Oct 30 17:11:40.409591 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404047 2572 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 17:11:40.409591 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404050 2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 17:11:40.409591 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404054 2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 17:11:40.409591 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404057 2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 17:11:40.409591 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404060 2572 flags.go:64] FLAG: --healthz-port="10248" Oct 30 17:11:40.409591 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404063 2572 flags.go:64] FLAG: --help="false" Oct 30 17:11:40.409591 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404066 2572 flags.go:64] FLAG: --hostname-override="ip-10-0-139-222.ec2.internal" Oct 30 17:11:40.409591 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404069 2572 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 17:11:40.409591 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404072 2572 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 17:11:40.409591 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404075 2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 17:11:40.409591 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404079 2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 17:11:40.409591 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404082 2572 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 17:11:40.409591 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404085 2572 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 17:11:40.409591 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404088 2572 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 17:11:40.410162 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404091 2572 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 17:11:40.410162 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404094 2572 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 17:11:40.410162 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404097 2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 17:11:40.410162 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404100 2572 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 17:11:40.410162 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404103 2572 flags.go:64] FLAG: --kube-reserved="" Oct 30 17:11:40.410162 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404106 2572 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 17:11:40.410162 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404109 2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 17:11:40.410162 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404112 2572 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 17:11:40.410162 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404115 2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 17:11:40.410162 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404118 2572 flags.go:64] FLAG: --lock-file="" Oct 30 17:11:40.410162 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404121 2572 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 17:11:40.410162 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404124 2572 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 17:11:40.410162 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404127 2572 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 17:11:40.410162 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404132 2572 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 17:11:40.410162 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404135 2572 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 17:11:40.410162 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404139 2572 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 17:11:40.410162 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404142 2572 flags.go:64] FLAG: --logging-format="text" Oct 30 17:11:40.410162 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404145 2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 17:11:40.410162 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404149 2572 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 17:11:40.410162 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404152 2572 flags.go:64] FLAG: --manifest-url="" Oct 30 17:11:40.410162 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404155 2572 flags.go:64] FLAG: --manifest-url-header="" Oct 30 17:11:40.410162 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404160 2572 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 17:11:40.410162 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404163 2572 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 17:11:40.410162 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404167 2572 flags.go:64] FLAG: --max-pods="110" Oct 30 17:11:40.410162 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404170 2572 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 17:11:40.410801 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404173 2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 17:11:40.410801 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404176 2572 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 17:11:40.410801 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404179 2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 17:11:40.410801 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404182 2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 17:11:40.410801 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404185 2572 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 17:11:40.410801 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404188 2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 17:11:40.410801 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404195 2572 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 17:11:40.410801 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404198 2572 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 17:11:40.410801 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404201 2572 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 17:11:40.410801 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404204 2572 flags.go:64] FLAG: --pod-cidr="" Oct 30 17:11:40.410801 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404207 2572 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 17:11:40.410801 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404210 2572 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 17:11:40.410801 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404213 2572 flags.go:64] FLAG: --pods-per-core="0" Oct 30 17:11:40.410801 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404215 2572 flags.go:64] FLAG: --port="10250" Oct 30 17:11:40.410801 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404218 2572 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 17:11:40.410801 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404221 2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d22446e589206b6a" Oct 30 17:11:40.410801 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404224 2572 flags.go:64] FLAG: --qos-reserved="" Oct 30 17:11:40.410801 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404227 2572 flags.go:64] FLAG: --read-only-port="10255" Oct 30 17:11:40.410801 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404230 2572 flags.go:64] FLAG: --register-node="true" Oct 30 17:11:40.410801 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404233 2572 flags.go:64] FLAG: --register-with-taints="" Oct 30 17:11:40.410801 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404237 2572 flags.go:64] FLAG: --registry-burst="10" Oct 30 17:11:40.410801 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404240 2572 flags.go:64] FLAG: --registry-qps="5" Oct 30 17:11:40.410801 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404245 2572 flags.go:64] FLAG: --reserved-cpus="" Oct 30 17:11:40.410801 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404247 2572 flags.go:64] FLAG: --reserved-memory="" Oct 30 17:11:40.410801 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404251 2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 17:11:40.410801 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404254 2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 17:11:40.411485 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404257 2572 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 17:11:40.411485 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404260 2572 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 17:11:40.411485 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404263 2572 flags.go:64] FLAG: --runonce="false" Oct 30 17:11:40.411485 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404266 2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 17:11:40.411485 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404269 2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 17:11:40.411485 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404272 2572 flags.go:64] FLAG: --seccomp-default="false" Oct 30 17:11:40.411485 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404275 2572 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 17:11:40.411485 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404277 2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 17:11:40.411485 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404281 2572 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 17:11:40.411485 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404286 2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 17:11:40.411485 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404289 2572 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 17:11:40.411485 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404293 2572 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 17:11:40.411485 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404296 2572 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 17:11:40.411485 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404299 2572 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 17:11:40.411485 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404302 2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 17:11:40.411485 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404305 2572 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 17:11:40.411485 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404308 2572 flags.go:64] FLAG: --system-cgroups="" Oct 30 17:11:40.411485 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404311 2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 17:11:40.411485 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404316 2572 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 17:11:40.411485 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404319 2572 flags.go:64] FLAG: --tls-cert-file="" Oct 30 17:11:40.411485 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404322 2572 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 17:11:40.411485 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404326 2572 flags.go:64] FLAG: --tls-min-version="" Oct 30 17:11:40.411485 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404328 2572 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 17:11:40.411485 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404331 2572 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 17:11:40.411485 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404334 2572 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 17:11:40.412109 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404337 2572 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 17:11:40.412109 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404340 2572 flags.go:64] FLAG: --v="2" Oct 30 17:11:40.412109 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404344 2572 flags.go:64] FLAG: --version="false" Oct 30 17:11:40.412109 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404350 2572 flags.go:64] FLAG: --vmodule="" Oct 30 17:11:40.412109 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404354 2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 17:11:40.412109 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404357 2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 17:11:40.412109 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404489 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:11:40.412109 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404493 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:11:40.412109 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404496 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:11:40.412109 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404498 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:11:40.412109 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404501 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:11:40.412109 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404504 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:11:40.412109 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404507 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:11:40.412109 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404509 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:11:40.412109 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404512 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:11:40.412109 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404514 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:11:40.412109 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404519 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:11:40.412109 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404522 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:11:40.412109 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404524 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:11:40.412109 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404527 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:11:40.412109 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404530 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:11:40.412616 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404532 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:11:40.412616 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404535 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:11:40.412616 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404538 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:11:40.412616 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404540 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:11:40.412616 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404543 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:11:40.412616 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404545 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:11:40.412616 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404548 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:11:40.412616 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404551 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:11:40.412616 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404553 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:11:40.412616 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404556 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:11:40.412616 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404558 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:11:40.412616 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404561 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:11:40.412616 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404564 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:11:40.412616 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404567 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:11:40.412616 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404571 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:11:40.412616 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404573 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:11:40.412616 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404576 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:11:40.412616 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404578 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:11:40.412616 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404581 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:11:40.413097 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404583 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:11:40.413097 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404586 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:11:40.413097 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404589 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:11:40.413097 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404591 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:11:40.413097 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404594 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:11:40.413097 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404597 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:11:40.413097 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404599 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:11:40.413097 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404602 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:11:40.413097 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404606 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:11:40.413097 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404609 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:11:40.413097 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404612 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:11:40.413097 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404614 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:11:40.413097 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404617 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:11:40.413097 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404621 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:11:40.413097 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404625 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:11:40.413097 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404627 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:11:40.413097 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404630 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:11:40.413097 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404633 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:11:40.413097 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404635 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:11:40.413581 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404638 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:11:40.413581 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404641 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:11:40.413581 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404643 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:11:40.413581 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404646 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:11:40.413581 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404648 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:11:40.413581 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404651 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:11:40.413581 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404654 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:11:40.413581 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404656 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:11:40.413581 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404659 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:11:40.413581 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404662 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:11:40.413581 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404665 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:11:40.413581 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404667 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:11:40.413581 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404670 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:11:40.413581 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404673 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:11:40.413581 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404675 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:11:40.413581 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404678 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:11:40.413581 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404680 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:11:40.413581 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404683 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:11:40.413581 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404685 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:11:40.413581 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404688 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:11:40.414087 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404691 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:11:40.414087 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404694 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:11:40.414087 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404697 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:11:40.414087 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404700 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:11:40.414087 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404702 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:11:40.414087 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404705 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:11:40.414087 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404708 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:11:40.414087 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404711 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:11:40.414087 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404714 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:11:40.414087 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404716 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:11:40.414087 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404719 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:11:40.414087 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404721 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:11:40.414087 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404725 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:11:40.414087 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404730 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:11:40.414087 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404733 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:11:40.414087 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404736 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:11:40.414087 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404739 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:11:40.414087 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404741 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:11:40.414087 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404744 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:11:40.414538 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404747 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:11:40.414538 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404749 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:11:40.414538 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404752 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:11:40.414538 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404754 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:11:40.414538 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404757 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:11:40.414538 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404760 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:11:40.414538 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404762 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:11:40.414538 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404765 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:11:40.414538 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404768 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:11:40.414538 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404770 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:11:40.414538 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404773 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:11:40.414538 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404775 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:11:40.414538 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404778 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:11:40.414538 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404781 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:11:40.414538 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.404808 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:11:40.414538 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404844 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:11:40.414992 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404850 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:11:40.414992 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404856 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:11:40.414992 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404870 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:11:40.414992 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404873 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:11:40.414992 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404876 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:11:40.414992 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404878 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:11:40.414992 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404881 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:11:40.414992 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404884 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:11:40.414992 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404886 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:11:40.414992 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404889 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:11:40.414992 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404892 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:11:40.414992 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404894 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:11:40.414992 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404897 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:11:40.414992 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404899 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:11:40.414992 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404902 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:11:40.414992 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404904 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:11:40.414992 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404916 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:11:40.414992 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404919 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:11:40.414992 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404922 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:11:40.414992 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404924 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:11:40.415454 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404927 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:11:40.415454 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404929 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:11:40.415454 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404932 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:11:40.415454 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404934 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:11:40.415454 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404937 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:11:40.415454 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404939 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:11:40.415454 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404942 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:11:40.415454 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404945 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:11:40.415454 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404947 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:11:40.415454 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404950 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:11:40.415454 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404952 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:11:40.415454 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404956 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:11:40.415454 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404959 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:11:40.415454 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404962 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:11:40.415454 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404965 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:11:40.415454 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404968 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:11:40.415454 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404970 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:11:40.415454 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404973 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:11:40.415454 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404977 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:11:40.415919 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404981 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:11:40.415919 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404984 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:11:40.415919 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404987 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:11:40.415919 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404990 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:11:40.415919 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404993 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:11:40.415919 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404995 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:11:40.415919 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.404998 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:11:40.415919 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405001 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:11:40.415919 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405003 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:11:40.415919 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405006 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:11:40.415919 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405008 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:11:40.415919 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405012 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:11:40.415919 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405014 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:11:40.415919 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405017 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:11:40.415919 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405019 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:11:40.415919 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405022 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:11:40.415919 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405024 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:11:40.415919 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405027 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:11:40.415919 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405029 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:11:40.415919 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405032 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:11:40.416414 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405035 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:11:40.416414 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405037 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:11:40.416414 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405040 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:11:40.416414 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405042 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:11:40.416414 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405046 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:11:40.416414 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405049 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:11:40.416414 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405052 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:11:40.416414 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405054 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:11:40.416414 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405057 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:11:40.416414 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405060 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:11:40.416414 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405063 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:11:40.416414 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405066 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:11:40.416414 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405068 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:11:40.416414 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405071 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:11:40.416414 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405074 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:11:40.416414 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405076 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:11:40.416414 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405079 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:11:40.416414 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405081 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:11:40.416414 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405084 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:11:40.416414 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405086 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:11:40.416918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405089 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:11:40.416918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405093 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:11:40.416918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405095 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:11:40.416918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405098 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:11:40.416918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405100 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:11:40.416918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405103 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:11:40.416918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405106 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:11:40.416918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405108 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:11:40.416918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405111 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:11:40.416918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405114 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:11:40.416918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405117 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:11:40.416918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405120 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:11:40.416918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405122 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:11:40.416918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405125 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:11:40.416918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405128 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:11:40.416918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405130 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:11:40.416918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405134 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:11:40.416918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405137 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:11:40.416918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405140 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:11:40.417402 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405142 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:11:40.417402 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405145 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:11:40.417402 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405148 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:11:40.417402 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405151 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:11:40.417402 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405153 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:11:40.417402 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405156 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:11:40.417402 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.405159 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:11:40.417402 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.412410 2572 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 17:11:40.417402 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.412425 2572 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 17:11:40.417402 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.412443 2572 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 17:11:40.417402 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.412448 2572 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 17:11:40.417402 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412503 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:11:40.417402 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412509 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:11:40.417402 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412512 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:11:40.417402 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412516 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:11:40.417402 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412520 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:11:40.417402 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412523 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:11:40.417402 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412526 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:11:40.417402 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412529 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:11:40.417880 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412532 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:11:40.417880 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412535 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:11:40.417880 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412538 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:11:40.417880 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412541 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:11:40.417880 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412543 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:11:40.417880 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412546 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:11:40.417880 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412549 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:11:40.417880 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412552 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:11:40.417880 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412555 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:11:40.417880 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412558 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:11:40.417880 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412560 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:11:40.417880 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412564 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:11:40.417880 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412567 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:11:40.417880 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412569 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:11:40.417880 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412572 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:11:40.417880 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412575 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:11:40.417880 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412578 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:11:40.417880 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412581 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:11:40.417880 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412583 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:11:40.417880 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412586 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:11:40.418353 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412588 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:11:40.418353 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412592 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:11:40.418353 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412595 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:11:40.418353 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412597 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:11:40.418353 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412600 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:11:40.418353 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412603 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:11:40.418353 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412605 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:11:40.418353 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412608 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:11:40.418353 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412611 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:11:40.418353 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412613 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:11:40.418353 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412616 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:11:40.418353 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412619 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:11:40.418353 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412622 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:11:40.418353 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412625 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:11:40.418353 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412627 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:11:40.418353 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412630 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:11:40.418353 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412633 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:11:40.418353 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412635 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:11:40.418353 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412638 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:11:40.418798 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412641 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:11:40.418798 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412644 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:11:40.418798 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412647 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:11:40.418798 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412650 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:11:40.418798 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412652 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:11:40.418798 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412655 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:11:40.418798 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412658 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:11:40.418798 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412660 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:11:40.418798 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412663 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:11:40.418798 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412667 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:11:40.418798 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412669 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:11:40.418798 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412672 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:11:40.418798 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412674 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:11:40.418798 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412677 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:11:40.418798 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412680 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:11:40.418798 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412683 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:11:40.418798 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412686 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:11:40.418798 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412688 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:11:40.418798 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412691 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:11:40.418798 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412694 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:11:40.419431 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412696 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:11:40.419431 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412699 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:11:40.419431 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412701 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:11:40.419431 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412704 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:11:40.419431 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412706 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:11:40.419431 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412709 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:11:40.419431 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412712 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:11:40.419431 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412715 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:11:40.419431 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412717 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:11:40.419431 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412720 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:11:40.419431 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412722 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:11:40.419431 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412725 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:11:40.419431 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412727 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:11:40.419431 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412730 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:11:40.419431 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412733 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:11:40.419431 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412736 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:11:40.419431 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412738 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:11:40.419431 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412741 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:11:40.419431 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412744 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:11:40.419873 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412747 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:11:40.419873 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412749 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:11:40.419873 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412752 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:11:40.419873 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412755 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:11:40.419873 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412757 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:11:40.419873 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412760 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:11:40.419873 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412763 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:11:40.419873 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412766 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:11:40.419873 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412769 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:11:40.419873 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412773 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:11:40.419873 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412776 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:11:40.419873 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412780 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:11:40.419873 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412782 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:11:40.419873 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412785 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:11:40.419873 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412788 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:11:40.419873 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412790 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:11:40.419873 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412793 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:11:40.419873 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412795 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:11:40.419873 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412798 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:11:40.420330 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412800 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:11:40.420330 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.412828 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:11:40.420330 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412993 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:11:40.420330 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.412998 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:11:40.420330 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413002 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:11:40.420330 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413004 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:11:40.420330 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413007 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:11:40.420330 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413010 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:11:40.420330 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413013 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:11:40.420330 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413015 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:11:40.420330 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413018 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:11:40.420330 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413021 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:11:40.420330 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413024 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:11:40.420330 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413027 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:11:40.420330 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413029 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:11:40.420330 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413032 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:11:40.420696 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413039 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:11:40.420696 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413047 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:11:40.420696 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413051 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:11:40.420696 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413054 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:11:40.420696 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413056 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:11:40.420696 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413059 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:11:40.420696 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413061 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:11:40.420696 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413064 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:11:40.420696 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413067 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:11:40.420696 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413069 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:11:40.420696 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413072 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:11:40.420696 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413074 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:11:40.420696 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413078 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:11:40.420696 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413081 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:11:40.420696 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413084 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:11:40.420696 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413086 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:11:40.420696 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413089 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:11:40.420696 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413092 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:11:40.420696 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413095 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:11:40.421172 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413098 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:11:40.421172 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413101 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:11:40.421172 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413104 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:11:40.421172 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413106 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:11:40.421172 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413109 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:11:40.421172 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413112 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:11:40.421172 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413115 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:11:40.421172 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413118 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:11:40.421172 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413120 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:11:40.421172 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413123 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:11:40.421172 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413126 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:11:40.421172 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413128 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:11:40.421172 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413131 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:11:40.421172 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413134 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:11:40.421172 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413136 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:11:40.421172 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413140 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:11:40.421172 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413142 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:11:40.421172 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413145 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:11:40.421172 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413149 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:11:40.421612 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413153 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:11:40.421612 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413156 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:11:40.421612 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413159 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:11:40.421612 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413162 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:11:40.421612 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413165 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:11:40.421612 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413167 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:11:40.421612 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413170 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:11:40.421612 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413172 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:11:40.421612 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413175 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:11:40.421612 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413177 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:11:40.421612 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413179 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:11:40.421612 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413182 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:11:40.421612 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413184 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:11:40.421612 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413187 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:11:40.421612 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413190 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:11:40.421612 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413192 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:11:40.421612 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413195 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:11:40.421612 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413198 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:11:40.421612 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413200 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:11:40.421612 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413203 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:11:40.422105 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413206 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:11:40.422105 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413209 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:11:40.422105 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413211 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:11:40.422105 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413214 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:11:40.422105 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413216 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:11:40.422105 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413218 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:11:40.422105 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413221 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:11:40.422105 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413224 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:11:40.422105 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413227 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:11:40.422105 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413230 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:11:40.422105 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413232 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:11:40.422105 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413235 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:11:40.422105 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413237 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:11:40.422105 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413240 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:11:40.422105 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413243 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:11:40.422105 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413245 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:11:40.422105 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413248 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:11:40.422105 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413250 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:11:40.422105 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413253 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:11:40.422105 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413255 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:11:40.422555 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413258 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:11:40.422555 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413260 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:11:40.422555 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413262 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:11:40.422555 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413265 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:11:40.422555 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413268 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:11:40.422555 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413270 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:11:40.422555 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413273 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:11:40.422555 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413275 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:11:40.422555 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413278 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:11:40.422555 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413281 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:11:40.422555 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413284 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:11:40.422555 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413286 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:11:40.422555 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413288 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:11:40.422555 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413291 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:11:40.422555 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.413314 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:11:40.422555 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413351 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:11:40.422978 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413355 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:11:40.422978 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413358 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:11:40.422978 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413361 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:11:40.422978 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413363 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:11:40.422978 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413366 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:11:40.422978 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413369 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:11:40.422978 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413372 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:11:40.422978 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413375 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:11:40.422978 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413377 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:11:40.422978 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413380 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:11:40.422978 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413382 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:11:40.422978 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413385 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:11:40.422978 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413387 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:11:40.422978 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413390 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:11:40.422978 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413392 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:11:40.422978 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413395 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:11:40.422978 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413398 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:11:40.422978 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413401 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:11:40.422978 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413403 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:11:40.422978 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413405 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:11:40.423439 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413410 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:11:40.423439 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413413 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:11:40.423439 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413415 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:11:40.423439 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413418 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:11:40.423439 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413420 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:11:40.423439 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413423 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:11:40.423439 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413426 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:11:40.423439 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413429 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:11:40.423439 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413431 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:11:40.423439 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413433 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:11:40.423439 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413436 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:11:40.423439 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413439 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:11:40.423439 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413441 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:11:40.423439 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413444 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:11:40.423439 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413446 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:11:40.423439 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413449 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:11:40.423439 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413451 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:11:40.423439 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413454 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:11:40.423439 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413456 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:11:40.423439 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413460 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:11:40.423918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413462 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:11:40.423918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413464 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:11:40.423918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413467 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:11:40.423918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413470 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:11:40.423918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413472 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:11:40.423918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413475 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:11:40.423918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413477 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:11:40.423918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413479 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:11:40.423918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413482 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:11:40.423918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413484 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:11:40.423918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413487 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:11:40.423918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413489 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:11:40.423918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413493 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:11:40.423918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413496 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:11:40.423918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413499 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:11:40.423918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413501 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:11:40.423918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413504 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:11:40.423918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413507 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:11:40.423918 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413510 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:11:40.424351 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413513 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:11:40.424351 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413515 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:11:40.424351 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413518 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:11:40.424351 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413520 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:11:40.424351 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413523 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:11:40.424351 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413525 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:11:40.424351 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413527 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:11:40.424351 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413530 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:11:40.424351 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413534 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:11:40.424351 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413538 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:11:40.424351 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413541 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:11:40.424351 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413543 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:11:40.424351 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413546 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:11:40.424351 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413549 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:11:40.424351 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413552 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:11:40.424351 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413555 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:11:40.424351 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413558 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:11:40.424351 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413560 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:11:40.424351 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413563 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:11:40.424794 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413565 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:11:40.424794 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413568 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:11:40.424794 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413570 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:11:40.424794 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413573 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:11:40.424794 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413575 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:11:40.424794 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413577 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:11:40.424794 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413581 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:11:40.424794 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413584 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:11:40.424794 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413587 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:11:40.424794 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413589 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:11:40.424794 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413592 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:11:40.424794 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413595 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:11:40.424794 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413598 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:11:40.424794 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413600 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:11:40.424794 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413604 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:11:40.424794 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413608 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:11:40.424794 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413611 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:11:40.424794 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413614 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:11:40.424794 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413618 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:11:40.425370 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413620 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:11:40.425370 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413623 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:11:40.425370 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413625 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:11:40.425370 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413628 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:11:40.425370 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413630 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:11:40.425370 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413633 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:11:40.425370 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413635 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:11:40.425370 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:40.413638 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:11:40.425370 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.413690 2572 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 17:11:40.425370 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.416164 2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 17:11:40.425370 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.417187 2572 server.go:1014] "Starting client certificate rotation" Oct 30 17:11:40.425370 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.417285 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 17:11:40.425370 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.417316 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 17:11:40.425370 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.419385 2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:11:40.426520 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.426496 2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:11:40.438496 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.438479 2572 log.go:25] "Validated CRI v1 runtime API" Oct 30 17:11:40.442420 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.442402 2572 log.go:25] "Validated CRI v1 image API" Oct 30 17:11:40.443292 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.443259 2572 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 17:11:40.449755 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.449728 2572 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8f1535e6-2162-42c8-a77d-ce4389de7ebe:/dev/nvme0n1p3 bf976311-5ed8-4095-90b2-5bbf64d61b63:/dev/nvme0n1p4] Oct 30 17:11:40.449807 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.449754 2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 17:11:40.455055 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.454932 2572 manager.go:217] Machine: {Timestamp:2026-10-30 17:11:40.453454612 +0000 UTC m=+0.377957156 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102519 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f0b6ac46ee1dc4d12f791bade63f8 SystemUUID:ec2f0b6a-c46e-e1dc-4d12-f791bade63f8 BootID:1a2be75d-e57e-424b-9ebb-9086aa8444ae Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f3:3a:c0:9d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f3:3a:c0:9d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:d7:56:08:e3:5f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 17:11:40.455055 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.455054 2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 17:11:40.455180 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.455165 2572 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 17:11:40.458266 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.458245 2572 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 17:11:40.458422 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.458267 2572 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-222.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 17:11:40.459144 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.459134 2572 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 17:11:40.459183 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.459146 2572 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 17:11:40.459183 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.459171 2572 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:11:40.459237 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.459183 2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:11:40.460456 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.460446 2572 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 17:11:40.464164 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.464150 2572 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 17:11:40.464289 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.464279 2572 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 17:11:40.465189 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.465178 2572 kubelet.go:493] "Attempting to sync node with API server" Oct 30 17:11:40.465241 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.465193 2572 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 17:11:40.465241 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.465207 2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 17:11:40.465241 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.465240 2572 kubelet.go:405] "Adding apiserver pod source" Oct 30 17:11:40.465342 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.465251 2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 17:11:40.465602 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.465590 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:11:40.467995 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.467972 2572 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 17:11:40.474796 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.474783 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:11:40.475726 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.475714 2572 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 17:11:40.475769 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.475745 2572 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 17:11:40.475951 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.475941 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 17:11:40.475982 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.475957 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 17:11:40.475982 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.475964 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 17:11:40.475982 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.475970 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 17:11:40.475982 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.475979 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 17:11:40.476086 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.475990 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 17:11:40.476589 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.476580 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 17:11:40.476630 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.476601 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 17:11:40.476630 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.476609 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 17:11:40.476630 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.476615 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 17:11:40.476630 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.476631 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 17:11:40.476856 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.476642 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 17:11:40.477376 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.477366 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 17:11:40.477427 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.477378 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 17:11:40.481206 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.481192 2572 server.go:1263] "Started kubelet" Oct 30 17:11:40.481329 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.481295 2572 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 17:11:40.481379 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.481294 2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 17:11:40.481379 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.481360 2572 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 17:11:40.482037 ip-10-0-139-222 systemd[1]: Started Kubernetes Kubelet. Oct 30 17:11:40.483208 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.483148 2572 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 17:11:40.484135 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.484118 2572 server.go:317] "Adding debug handlers to kubelet server" Oct 30 17:11:40.488899 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:40.488873 2572 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 17:11:40.493919 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.493901 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 17:11:40.494624 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.494600 2572 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 17:11:40.494720 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:40.494703 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-222.ec2.internal\" not found" Oct 30 17:11:40.495453 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.495435 2572 factory.go:55] Registering systemd factory Oct 30 17:11:40.495453 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.495452 2572 factory.go:223] Registration of the systemd container factory successfully Oct 30 17:11:40.495584 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.495434 2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 17:11:40.495584 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.495437 2572 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 17:11:40.495584 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.495497 2572 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 17:11:40.495584 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.495546 2572 reconstruct.go:97] "Volume reconstruction finished" Oct 30 17:11:40.495584 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.495556 2572 reconciler.go:29] "Reconciler: start to sync state" Oct 30 17:11:40.495822 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.495667 2572 factory.go:153] Registering CRI-O factory Oct 30 17:11:40.495822 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.495683 2572 factory.go:223] Registration of the crio container factory successfully Oct 30 17:11:40.495822 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.495735 2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 17:11:40.495822 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.495761 2572 factory.go:103] Registering Raw factory Oct 30 17:11:40.495822 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.495773 2572 manager.go:1199] Started watching for new ooms in manager Oct 30 17:11:40.496772 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.496760 2572 manager.go:319] Starting recovery of all containers Oct 30 17:11:40.506977 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.506960 2572 manager.go:324] Recovery completed Oct 30 17:11:40.510837 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.510825 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:11:40.512188 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.512168 2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 17:11:40.512466 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.512450 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-222.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:11:40.512534 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.512478 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-222.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:11:40.512534 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.512489 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-222.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:11:40.512986 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.512975 2572 cpu_manager.go:226] "Starting" policy="none" Oct 30 17:11:40.512986 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.512984 2572 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 17:11:40.513110 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.513000 2572 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 17:11:40.513110 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.513049 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:11:40.513679 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.513664 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-222.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:11:40.513726 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.513690 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-222.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:11:40.513726 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.513699 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-222.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:11:40.513993 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.513982 2572 policy_none.go:50] "Start" Oct 30 17:11:40.514028 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.513997 2572 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 17:11:40.514028 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.514006 2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 17:11:40.514855 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.514844 2572 policy_none.go:44] "Start" Oct 30 17:11:40.518885 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.518849 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-222.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 17:11:40.518885 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:40.518854 2572 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-222.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 17:11:40.519998 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:40.518854 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-222.ec2.internal.18e35de1ad5cfab9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-222.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-222.ec2.internal,},FirstTimestamp:2026-10-30 17:11:40.481166009 +0000 UTC m=+0.405668554,LastTimestamp:2026-10-30 17:11:40.481166009 +0000 UTC m=+0.405668554,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-222.ec2.internal,}" Oct 30 17:11:40.536940 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:40.536408 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-222.ec2.internal.18e35de1af3a93d6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-222.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-222.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-222.ec2.internal,},FirstTimestamp:2026-10-30 17:11:40.512465878 +0000 UTC m=+0.436968423,LastTimestamp:2026-10-30 17:11:40.512465878 +0000 UTC m=+0.436968423,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-222.ec2.internal,}" Oct 30 17:11:40.536940 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:40.536712 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 17:11:40.536940 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:40.536796 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 17:11:40.540548 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.540519 2572 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 17:11:40.543022 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.543006 2572 manager.go:341] "Starting Device Plugin manager" Oct 30 17:11:40.543736 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:40.543716 2572 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 17:11:40.543771 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.543742 2572 server.go:87] "Starting device plugin registration server" Oct 30 17:11:40.544036 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.544025 2572 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 17:11:40.544067 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.544041 2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 17:11:40.544116 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.544102 2572 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 17:11:40.544260 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.544186 2572 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 17:11:40.544260 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.544195 2572 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 17:11:40.544664 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:40.544629 2572 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 17:11:40.544720 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:40.544677 2572 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-222.ec2.internal\" not found" Oct 30 17:11:40.548300 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:40.548281 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-222.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 17:11:40.549587 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.549570 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k4htm" Oct 30 17:11:40.561403 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.561387 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k4htm" Oct 30 17:11:40.565235 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:40.565175 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-222.ec2.internal.18e35de1af3ad56d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-222.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-222.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-222.ec2.internal,},FirstTimestamp:2026-10-30 17:11:40.512482669 +0000 UTC m=+0.436985214,LastTimestamp:2026-10-30 17:11:40.512482669 +0000 UTC m=+0.436985214,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-222.ec2.internal,}" Oct 30 17:11:40.620739 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.620716 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 17:11:40.621816 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.621802 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 17:11:40.621882 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.621823 2572 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 17:11:40.621882 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.621841 2572 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 17:11:40.621980 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:40.621893 2572 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 17:11:40.635135 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.635118 2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:11:40.644395 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.644383 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:11:40.645379 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.645336 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-222.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:11:40.645379 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.645366 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-222.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:11:40.645515 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.645383 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-222.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:11:40.645515 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.645411 2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-222.ec2.internal" Oct 30 17:11:40.664838 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.664822 2572 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-222.ec2.internal" Oct 30 17:11:40.664923 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:40.664844 2572 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-222.ec2.internal\": node \"ip-10-0-139-222.ec2.internal\" not found" Oct 30 17:11:40.712067 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:40.712038 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-222.ec2.internal\" not found" Oct 30 17:11:40.723010 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.722986 2572 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-222.ec2.internal"] Oct 30 17:11:40.723100 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.723051 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:11:40.723940 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.723924 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-222.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:11:40.724024 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.723952 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-222.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:11:40.724024 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.723964 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-222.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:11:40.725238 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.725226 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:11:40.725396 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.725380 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal" Oct 30 17:11:40.725461 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.725417 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:11:40.725986 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.725969 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-222.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:11:40.726068 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.725985 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-222.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:11:40.726068 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.725995 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-222.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:11:40.726068 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.726006 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-222.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:11:40.726068 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.726008 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-222.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:11:40.726068 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.726020 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-222.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:11:40.726974 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.726962 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-222.ec2.internal" Oct 30 17:11:40.727027 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.726994 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:11:40.727651 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.727635 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-222.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:11:40.727724 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.727663 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-222.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:11:40.727724 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.727680 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-222.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:11:40.758534 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:40.758502 2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-222.ec2.internal\" not found" node="ip-10-0-139-222.ec2.internal" Oct 30 17:11:40.761048 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:40.761030 2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-222.ec2.internal\" not found" node="ip-10-0-139-222.ec2.internal" Oct 30 17:11:40.798173 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.798158 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d961f94773b0889650472ea1a7f54825-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal\" (UID: \"d961f94773b0889650472ea1a7f54825\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal" Oct 30 17:11:40.798263 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.798181 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d961f94773b0889650472ea1a7f54825-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal\" (UID: \"d961f94773b0889650472ea1a7f54825\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal" Oct 30 17:11:40.798263 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.798198 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/67161c7401d4577fb0911752be0668c2-config\") pod \"kube-apiserver-proxy-ip-10-0-139-222.ec2.internal\" (UID: \"67161c7401d4577fb0911752be0668c2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-222.ec2.internal" Oct 30 17:11:40.812836 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:40.812818 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-222.ec2.internal\" not found" Oct 30 17:11:40.898762 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.898701 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/67161c7401d4577fb0911752be0668c2-config\") pod \"kube-apiserver-proxy-ip-10-0-139-222.ec2.internal\" (UID: \"67161c7401d4577fb0911752be0668c2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-222.ec2.internal" Oct 30 17:11:40.898762 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.898733 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d961f94773b0889650472ea1a7f54825-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal\" (UID: \"d961f94773b0889650472ea1a7f54825\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal" Oct 30 17:11:40.898762 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.898755 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d961f94773b0889650472ea1a7f54825-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal\" (UID: \"d961f94773b0889650472ea1a7f54825\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal" Oct 30 17:11:40.898957 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.898811 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d961f94773b0889650472ea1a7f54825-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal\" (UID: \"d961f94773b0889650472ea1a7f54825\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal" Oct 30 17:11:40.898957 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.898838 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d961f94773b0889650472ea1a7f54825-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal\" (UID: \"d961f94773b0889650472ea1a7f54825\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal" Oct 30 17:11:40.898957 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:40.898813 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/67161c7401d4577fb0911752be0668c2-config\") pod \"kube-apiserver-proxy-ip-10-0-139-222.ec2.internal\" (UID: \"67161c7401d4577fb0911752be0668c2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-222.ec2.internal" Oct 30 17:11:40.913032 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:40.913015 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-222.ec2.internal\" not found" Oct 30 17:11:41.013255 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:41.013221 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-222.ec2.internal\" not found" Oct 30 17:11:41.063399 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:41.063380 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal" Oct 30 17:11:41.065525 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:41.065509 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-222.ec2.internal" Oct 30 17:11:41.113857 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:41.113713 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-222.ec2.internal\" not found" Oct 30 17:11:41.214213 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:41.214168 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-222.ec2.internal\" not found" Oct 30 17:11:41.314718 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:41.314692 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-222.ec2.internal\" not found" Oct 30 17:11:41.415072 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:41.415038 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-222.ec2.internal\" not found" Oct 30 17:11:41.417153 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:41.417129 2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 17:11:41.417277 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:41.417259 2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 17:11:41.515436 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:41.515350 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-222.ec2.internal\" not found" Oct 30 17:11:41.566990 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:41.566952 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 17:06:41 +0000 UTC" deadline="2028-06-21 21:48:15.171515605 +0000 UTC" Oct 30 17:11:41.566990 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:41.566984 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14404h36m33.604535785s" Oct 30 17:11:41.604220 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:41.604200 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:11:41.606361 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:41.606341 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-222.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:11:41.606439 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:41.606374 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-222.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:11:41.606439 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:41.606385 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-222.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:11:41.616733 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:41.616526 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-222.ec2.internal\" not found" Oct 30 17:11:41.657536 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:41.657500 2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:11:41.669137 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:41.669113 2572 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:11:41.695195 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:41.695166 2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal" Oct 30 17:11:41.723201 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:41.723184 2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:11:41.723328 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:41.723239 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67161c7401d4577fb0911752be0668c2.slice/crio-b1f4230704e571ae81b3c38071fcfbdcdf9ff5933438e5bdbb3e99520e76bcee WatchSource:0}: Error finding container b1f4230704e571ae81b3c38071fcfbdcdf9ff5933438e5bdbb3e99520e76bcee: Status 404 returned error can't find the container with id b1f4230704e571ae81b3c38071fcfbdcdf9ff5933438e5bdbb3e99520e76bcee Oct 30 17:11:41.723757 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:41.723734 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd961f94773b0889650472ea1a7f54825.slice/crio-8860d3615245d0a04f931cc48968445461dce11d162d2a8665fd127bddd7c6ae WatchSource:0}: Error finding container 8860d3615245d0a04f931cc48968445461dce11d162d2a8665fd127bddd7c6ae: Status 404 returned error can't find the container with id 8860d3615245d0a04f931cc48968445461dce11d162d2a8665fd127bddd7c6ae Oct 30 17:11:41.725021 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:41.725006 2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-222.ec2.internal" Oct 30 17:11:41.729122 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:41.728709 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:11:41.744412 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:41.744394 2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:11:41.820163 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:41.820086 2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:11:42.466994 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.466947 2572 apiserver.go:52] "Watching apiserver" Oct 30 17:11:42.496598 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.496536 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 17:11:42.510688 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.510654 2572 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 17:11:42.511008 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.510989 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-db6jk","openshift-network-diagnostics/network-check-target-t2pqm","kube-system/konnectivity-agent-jnfks","openshift-cluster-node-tuning-operator/tuned-pvpvp","openshift-dns/node-resolver-zgb2b","openshift-multus/network-metrics-daemon-87jjv","kube-system/global-pull-secret-syncer-5mtjr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal","openshift-multus/multus-additional-cni-plugins-7gvrp","openshift-network-operator/iptables-alerter-pj5s7","kube-system/kube-apiserver-proxy-ip-10-0-139-222.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq","openshift-image-registry/node-ca-stqrx","openshift-ovn-kubernetes/ovnkube-node-w54kt"] Oct 30 17:11:42.512646 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.512593 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:11:42.512776 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:42.512675 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mtjr" podUID="2906e95e-3baa-4c79-ad9c-b2592c1d414f" Oct 30 17:11:42.514446 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.514423 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-stqrx" Oct 30 17:11:42.515906 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.515881 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jnfks" Oct 30 17:11:42.518015 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.517985 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.518652 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.518630 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zgb2b" Oct 30 17:11:42.520196 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.520173 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:11:42.520382 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:42.520313 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jjv" podUID="6178534d-9f75-49c7-a977-de65737cbcea" Oct 30 17:11:42.521543 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.521497 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:11:42.522863 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.522518 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.524250 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.524231 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t2pqm" Oct 30 17:11:42.524346 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:42.524302 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t2pqm" podUID="d39ddd3a-781f-4084-b4e2-624b130c1025" Oct 30 17:11:42.526748 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.526694 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pj5s7" Oct 30 17:11:42.528155 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.528136 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.529720 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.529692 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:11:42.530432 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.530410 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 17:11:42.530593 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.530562 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 17:11:42.530682 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.530613 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 17:11:42.530682 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.530672 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-49f6v\"" Oct 30 17:11:42.533164 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.532971 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 17:11:42.533164 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.533140 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9lrx5\"" Oct 30 17:11:42.534580 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.533981 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 17:11:42.534580 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.534076 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:11:42.534580 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.534553 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qn7sb\"" Oct 30 17:11:42.539209 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.538920 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mkzvv\"" Oct 30 17:11:42.539209 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.539011 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2wbm5\"" Oct 30 17:11:42.539209 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.539053 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 17:11:42.539209 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.539087 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 17:11:42.539558 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.539452 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8vxcg\"" Oct 30 17:11:42.539558 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.539535 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 17:11:42.539661 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.539594 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 17:11:42.539661 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.539641 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 17:11:42.539767 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.539689 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 17:11:42.539820 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.539781 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 17:11:42.540454 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.540365 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 17:11:42.541600 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.541581 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 17:11:42.541885 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.541856 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:11:42.544073 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.544055 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 17:11:42.544256 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.544082 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 17:11:42.544394 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.544194 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-g55vh\"" Oct 30 17:11:42.545716 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.545692 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 17:11:42.545818 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.545745 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 17:11:42.545898 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.545835 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n8szd\"" Oct 30 17:11:42.547718 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.547696 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ctbv7\"" Oct 30 17:11:42.547821 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.547783 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 17:11:42.547821 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.547814 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 17:11:42.553369 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.553349 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 17:11:42.553469 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.553398 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 17:11:42.553563 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.553545 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 17:11:42.553725 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.553643 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 17:11:42.553896 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.553678 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 17:11:42.558600 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.558529 2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 17:11:42.558716 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.558623 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xt7v2" Oct 30 17:11:42.597269 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.597220 2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 17:11:42.607015 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.606957 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2664a881-44d7-470a-93d0-df1b3eaca0bc-serviceca\") pod \"node-ca-stqrx\" (UID: \"2664a881-44d7-470a-93d0-df1b3eaca0bc\") " pod="openshift-image-registry/node-ca-stqrx" Oct 30 17:11:42.607015 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607000 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd6ddeda-aec9-4872-9e43-084577cfcd7d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:11:42.607194 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607028 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r67qt\" (UniqueName: \"kubernetes.io/projected/cd6ddeda-aec9-4872-9e43-084577cfcd7d-kube-api-access-r67qt\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:11:42.607194 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607054 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55f62602-c7b7-4a6c-b244-f6e48775733f-cni-binary-copy\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.607194 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607083 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f608efeb-1bda-4375-a428-c6db6341fd0b-ovnkube-script-lib\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.607194 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607109 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f442e60f-8ccf-485e-9709-60a909d482bc-konnectivity-ca\") pod \"konnectivity-agent-jnfks\" (UID: \"f442e60f-8ccf-485e-9709-60a909d482bc\") " pod="kube-system/konnectivity-agent-jnfks" Oct 30 17:11:42.607194 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607134 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/467187ee-7039-4bb4-adbe-38161ee7c336-hosts-file\") pod \"node-resolver-zgb2b\" (UID: \"467187ee-7039-4bb4-adbe-38161ee7c336\") " pod="openshift-dns/node-resolver-zgb2b" Oct 30 17:11:42.607194 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607165 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/467187ee-7039-4bb4-adbe-38161ee7c336-tmp-dir\") pod \"node-resolver-zgb2b\" (UID: \"467187ee-7039-4bb4-adbe-38161ee7c336\") " pod="openshift-dns/node-resolver-zgb2b" Oct 30 17:11:42.607550 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607190 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6178534d-9f75-49c7-a977-de65737cbcea-metrics-certs\") pod \"network-metrics-daemon-87jjv\" (UID: \"6178534d-9f75-49c7-a977-de65737cbcea\") " pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:11:42.607550 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607241 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d05e04a8-3ddf-4261-add0-c2324c6c9e21-tmp\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.607550 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607263 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-var-lib-kubelet\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.607550 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607278 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-node-log\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.607550 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607293 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-log-socket\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.607550 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607314 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-run-netns\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.607550 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607338 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/55f62602-c7b7-4a6c-b244-f6e48775733f-multus-daemon-config\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.607550 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607356 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-run-systemd\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.607550 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607372 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5mmm\" (UniqueName: \"kubernetes.io/projected/2664a881-44d7-470a-93d0-df1b3eaca0bc-kube-api-access-p5mmm\") pod \"node-ca-stqrx\" (UID: \"2664a881-44d7-470a-93d0-df1b3eaca0bc\") " pod="openshift-image-registry/node-ca-stqrx" Oct 30 17:11:42.607550 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607389 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-host\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.607550 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607406 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn8lc\" (UniqueName: \"kubernetes.io/projected/d05e04a8-3ddf-4261-add0-c2324c6c9e21-kube-api-access-jn8lc\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.607550 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607436 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-multus-socket-dir-parent\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.607550 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607450 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-hostroot\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.607550 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607465 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.607550 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607487 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2906e95e-3baa-4c79-ad9c-b2592c1d414f-dbus\") pod \"global-pull-secret-syncer-5mtjr\" (UID: \"2906e95e-3baa-4c79-ad9c-b2592c1d414f\") " pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:11:42.607550 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607503 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2664a881-44d7-470a-93d0-df1b3eaca0bc-host\") pod \"node-ca-stqrx\" (UID: \"2664a881-44d7-470a-93d0-df1b3eaca0bc\") " pod="openshift-image-registry/node-ca-stqrx" Oct 30 17:11:42.607550 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607525 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-multus-cni-dir\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.608448 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607542 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-run-k8s-cni-cncf-io\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.608448 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607557 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-systemd-units\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.608448 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607572 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-run-ovn-kubernetes\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.608448 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607587 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-cni-netd\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.608448 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607602 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f608efeb-1bda-4375-a428-c6db6341fd0b-ovn-node-metrics-cert\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.608448 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607617 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2906e95e-3baa-4c79-ad9c-b2592c1d414f-kubelet-config\") pod \"global-pull-secret-syncer-5mtjr\" (UID: \"2906e95e-3baa-4c79-ad9c-b2592c1d414f\") " pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:11:42.608448 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607640 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-cnibin\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.608448 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607656 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-etc-kubernetes\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.608448 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607672 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9swv\" (UniqueName: \"kubernetes.io/projected/55f62602-c7b7-4a6c-b244-f6e48775733f-kube-api-access-m9swv\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.608448 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607700 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-run-netns\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.608448 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607720 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-registration-dir\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:11:42.608448 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607741 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqrt8\" (UniqueName: \"kubernetes.io/projected/6178534d-9f75-49c7-a977-de65737cbcea-kube-api-access-nqrt8\") pod \"network-metrics-daemon-87jjv\" (UID: \"6178534d-9f75-49c7-a977-de65737cbcea\") " pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:11:42.608448 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607767 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:11:42.608448 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607789 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs2g4\" (UniqueName: \"kubernetes.io/projected/c7b7edd7-11cc-4fac-92ae-a08140056adb-kube-api-access-xs2g4\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:11:42.608448 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607805 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-sysconfig\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.608448 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607822 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd6ddeda-aec9-4872-9e43-084577cfcd7d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:11:42.609069 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607838 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-system-cni-dir\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.609069 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607852 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-multus-conf-dir\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.609069 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607867 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-var-lib-openvswitch\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.609069 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607883 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-kubernetes\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.609069 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607901 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-run\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.609069 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607925 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-cni-bin\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.609069 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607955 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2906e95e-3baa-4c79-ad9c-b2592c1d414f-original-pull-secret\") pod \"global-pull-secret-syncer-5mtjr\" (UID: \"2906e95e-3baa-4c79-ad9c-b2592c1d414f\") " pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:11:42.609069 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607972 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sh5q\" (UniqueName: \"kubernetes.io/projected/933be7f4-8d62-4735-807f-7aa1439583fa-kube-api-access-6sh5q\") pod \"iptables-alerter-pj5s7\" (UID: \"933be7f4-8d62-4735-807f-7aa1439583fa\") " pod="openshift-network-operator/iptables-alerter-pj5s7" Oct 30 17:11:42.609069 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.607997 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-slash\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.609069 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.608016 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f608efeb-1bda-4375-a428-c6db6341fd0b-ovnkube-config\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.609069 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.608040 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d49vx\" (UniqueName: \"kubernetes.io/projected/f608efeb-1bda-4375-a428-c6db6341fd0b-kube-api-access-d49vx\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.609069 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.608059 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-etc-selinux\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:11:42.609069 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.608074 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-sys-fs\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:11:42.609069 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.608096 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-tuned\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.609069 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.608114 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd6ddeda-aec9-4872-9e43-084577cfcd7d-system-cni-dir\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:11:42.609069 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.608137 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd6ddeda-aec9-4872-9e43-084577cfcd7d-cnibin\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:11:42.609623 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.608155 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd6ddeda-aec9-4872-9e43-084577cfcd7d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:11:42.609623 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.608172 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/933be7f4-8d62-4735-807f-7aa1439583fa-iptables-alerter-script\") pod \"iptables-alerter-pj5s7\" (UID: \"933be7f4-8d62-4735-807f-7aa1439583fa\") " pod="openshift-network-operator/iptables-alerter-pj5s7" Oct 30 17:11:42.609623 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.608187 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-etc-openvswitch\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.609623 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.608202 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-run-ovn\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.609623 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.608217 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-sysctl-conf\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.609623 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.608234 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-systemd\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.609623 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.608249 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-var-lib-cni-bin\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.609623 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.608265 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-run-openvswitch\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.609623 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.608300 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f442e60f-8ccf-485e-9709-60a909d482bc-agent-certs\") pod \"konnectivity-agent-jnfks\" (UID: \"f442e60f-8ccf-485e-9709-60a909d482bc\") " pod="kube-system/konnectivity-agent-jnfks" Oct 30 17:11:42.609623 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.608321 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7n64\" (UniqueName: \"kubernetes.io/projected/467187ee-7039-4bb4-adbe-38161ee7c336-kube-api-access-p7n64\") pod \"node-resolver-zgb2b\" (UID: \"467187ee-7039-4bb4-adbe-38161ee7c336\") " pod="openshift-dns/node-resolver-zgb2b" Oct 30 17:11:42.609623 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.608337 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-device-dir\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:11:42.609623 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.608355 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-var-lib-kubelet\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.609623 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.608371 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-var-lib-cni-multus\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.609623 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.608387 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-run-multus-certs\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.609623 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.608402 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/933be7f4-8d62-4735-807f-7aa1439583fa-host-slash\") pod \"iptables-alerter-pj5s7\" (UID: \"933be7f4-8d62-4735-807f-7aa1439583fa\") " pod="openshift-network-operator/iptables-alerter-pj5s7" Oct 30 17:11:42.609623 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.608418 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-modprobe-d\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.610330 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.608432 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-sys\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.610330 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.608447 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-lib-modules\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.610330 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.608462 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-socket-dir\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:11:42.610330 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.608478 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-sysctl-d\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.610330 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.608495 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd6ddeda-aec9-4872-9e43-084577cfcd7d-cni-binary-copy\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:11:42.610330 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.608510 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-kubelet\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.610330 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.608525 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f608efeb-1bda-4375-a428-c6db6341fd0b-env-overrides\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.610330 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.608544 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4njs\" (UniqueName: \"kubernetes.io/projected/d39ddd3a-781f-4084-b4e2-624b130c1025-kube-api-access-z4njs\") pod \"network-check-target-t2pqm\" (UID: \"d39ddd3a-781f-4084-b4e2-624b130c1025\") " pod="openshift-network-diagnostics/network-check-target-t2pqm" Oct 30 17:11:42.634413 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.634323 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal" event={"ID":"d961f94773b0889650472ea1a7f54825","Type":"ContainerStarted","Data":"8860d3615245d0a04f931cc48968445461dce11d162d2a8665fd127bddd7c6ae"} Oct 30 17:11:42.636046 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.636017 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-222.ec2.internal" event={"ID":"67161c7401d4577fb0911752be0668c2","Type":"ContainerStarted","Data":"b1f4230704e571ae81b3c38071fcfbdcdf9ff5933438e5bdbb3e99520e76bcee"} Oct 30 17:11:42.709501 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.709461 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d49vx\" (UniqueName: \"kubernetes.io/projected/f608efeb-1bda-4375-a428-c6db6341fd0b-kube-api-access-d49vx\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.709665 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.709506 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-etc-selinux\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:11:42.709665 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.709535 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-sys-fs\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:11:42.709665 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.709561 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-tuned\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.709665 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.709586 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd6ddeda-aec9-4872-9e43-084577cfcd7d-system-cni-dir\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:11:42.709665 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.709609 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd6ddeda-aec9-4872-9e43-084577cfcd7d-cnibin\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:11:42.709665 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.709661 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd6ddeda-aec9-4872-9e43-084577cfcd7d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:11:42.709958 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.709686 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/933be7f4-8d62-4735-807f-7aa1439583fa-iptables-alerter-script\") pod \"iptables-alerter-pj5s7\" (UID: \"933be7f4-8d62-4735-807f-7aa1439583fa\") " pod="openshift-network-operator/iptables-alerter-pj5s7" Oct 30 17:11:42.709958 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.709710 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-etc-openvswitch\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.709958 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.709736 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-run-ovn\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.709958 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.709760 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-sysctl-conf\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.709958 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.709783 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-systemd\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.709958 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.709816 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-var-lib-cni-bin\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.709958 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.709841 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-run-openvswitch\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.709958 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.709865 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f442e60f-8ccf-485e-9709-60a909d482bc-agent-certs\") pod \"konnectivity-agent-jnfks\" (UID: \"f442e60f-8ccf-485e-9709-60a909d482bc\") " pod="kube-system/konnectivity-agent-jnfks" Oct 30 17:11:42.709958 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.709888 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7n64\" (UniqueName: \"kubernetes.io/projected/467187ee-7039-4bb4-adbe-38161ee7c336-kube-api-access-p7n64\") pod \"node-resolver-zgb2b\" (UID: \"467187ee-7039-4bb4-adbe-38161ee7c336\") " pod="openshift-dns/node-resolver-zgb2b" Oct 30 17:11:42.709958 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.709937 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-device-dir\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:11:42.710262 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.709961 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-var-lib-kubelet\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.710262 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.709994 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-var-lib-cni-multus\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.710262 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.709995 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd6ddeda-aec9-4872-9e43-084577cfcd7d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:11:42.710262 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710041 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-run-multus-certs\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.710262 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710069 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/933be7f4-8d62-4735-807f-7aa1439583fa-host-slash\") pod \"iptables-alerter-pj5s7\" (UID: \"933be7f4-8d62-4735-807f-7aa1439583fa\") " pod="openshift-network-operator/iptables-alerter-pj5s7" Oct 30 17:11:42.710262 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710086 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-etc-openvswitch\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.710262 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710100 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-modprobe-d\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.710262 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710128 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-run-ovn\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.710262 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710129 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-sys\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.710262 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710165 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-lib-modules\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.710262 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710177 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-sys\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.710262 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710193 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-socket-dir\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:11:42.710262 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710222 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-sysctl-d\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.710262 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710227 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-etc-selinux\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:11:42.710262 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710249 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd6ddeda-aec9-4872-9e43-084577cfcd7d-cni-binary-copy\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:11:42.710814 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710272 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-sys-fs\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:11:42.710814 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710276 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-kubelet\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.710814 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710307 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f608efeb-1bda-4375-a428-c6db6341fd0b-env-overrides\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.710814 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710311 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-kubelet\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.710814 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710335 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4njs\" (UniqueName: \"kubernetes.io/projected/d39ddd3a-781f-4084-b4e2-624b130c1025-kube-api-access-z4njs\") pod \"network-check-target-t2pqm\" (UID: \"d39ddd3a-781f-4084-b4e2-624b130c1025\") " pod="openshift-network-diagnostics/network-check-target-t2pqm" Oct 30 17:11:42.710814 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710357 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2664a881-44d7-470a-93d0-df1b3eaca0bc-serviceca\") pod \"node-ca-stqrx\" (UID: \"2664a881-44d7-470a-93d0-df1b3eaca0bc\") " pod="openshift-image-registry/node-ca-stqrx" Oct 30 17:11:42.710814 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710378 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd6ddeda-aec9-4872-9e43-084577cfcd7d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:11:42.710814 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710403 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r67qt\" (UniqueName: \"kubernetes.io/projected/cd6ddeda-aec9-4872-9e43-084577cfcd7d-kube-api-access-r67qt\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:11:42.710814 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710427 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55f62602-c7b7-4a6c-b244-f6e48775733f-cni-binary-copy\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.710814 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710438 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-sysctl-conf\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.710814 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710453 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f608efeb-1bda-4375-a428-c6db6341fd0b-ovnkube-script-lib\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.710814 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710480 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f442e60f-8ccf-485e-9709-60a909d482bc-konnectivity-ca\") pod \"konnectivity-agent-jnfks\" (UID: \"f442e60f-8ccf-485e-9709-60a909d482bc\") " pod="kube-system/konnectivity-agent-jnfks" Oct 30 17:11:42.710814 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710484 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-systemd\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.710814 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710507 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/467187ee-7039-4bb4-adbe-38161ee7c336-hosts-file\") pod \"node-resolver-zgb2b\" (UID: \"467187ee-7039-4bb4-adbe-38161ee7c336\") " pod="openshift-dns/node-resolver-zgb2b" Oct 30 17:11:42.710814 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710525 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-var-lib-cni-bin\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.710814 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710535 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/467187ee-7039-4bb4-adbe-38161ee7c336-tmp-dir\") pod \"node-resolver-zgb2b\" (UID: \"467187ee-7039-4bb4-adbe-38161ee7c336\") " pod="openshift-dns/node-resolver-zgb2b" Oct 30 17:11:42.710814 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710564 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-run-openvswitch\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.711403 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710562 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6178534d-9f75-49c7-a977-de65737cbcea-metrics-certs\") pod \"network-metrics-daemon-87jjv\" (UID: \"6178534d-9f75-49c7-a977-de65737cbcea\") " pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:11:42.711403 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710601 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d05e04a8-3ddf-4261-add0-c2324c6c9e21-tmp\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.711403 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710624 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-var-lib-kubelet\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.711403 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710649 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-node-log\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.711403 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:42.710656 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:11:42.711403 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710674 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-log-socket\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.711403 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710703 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-run-netns\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.711403 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:42.710726 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6178534d-9f75-49c7-a977-de65737cbcea-metrics-certs podName:6178534d-9f75-49c7-a977-de65737cbcea nodeName:}" failed. No retries permitted until 2026-10-30 17:11:43.210703051 +0000 UTC m=+3.135205603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6178534d-9f75-49c7-a977-de65737cbcea-metrics-certs") pod "network-metrics-daemon-87jjv" (UID: "6178534d-9f75-49c7-a977-de65737cbcea") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:11:42.711403 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710739 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-run-netns\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.711403 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710754 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/55f62602-c7b7-4a6c-b244-f6e48775733f-multus-daemon-config\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.711403 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710782 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-run-systemd\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.711403 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710808 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5mmm\" (UniqueName: \"kubernetes.io/projected/2664a881-44d7-470a-93d0-df1b3eaca0bc-kube-api-access-p5mmm\") pod \"node-ca-stqrx\" (UID: \"2664a881-44d7-470a-93d0-df1b3eaca0bc\") " pod="openshift-image-registry/node-ca-stqrx" Oct 30 17:11:42.711403 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710836 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-host\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.711403 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710861 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn8lc\" (UniqueName: \"kubernetes.io/projected/d05e04a8-3ddf-4261-add0-c2324c6c9e21-kube-api-access-jn8lc\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.711403 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710888 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-multus-socket-dir-parent\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.711403 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710914 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-hostroot\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.711403 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710967 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.712065 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710998 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2906e95e-3baa-4c79-ad9c-b2592c1d414f-dbus\") pod \"global-pull-secret-syncer-5mtjr\" (UID: \"2906e95e-3baa-4c79-ad9c-b2592c1d414f\") " pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:11:42.712065 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.711022 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2664a881-44d7-470a-93d0-df1b3eaca0bc-host\") pod \"node-ca-stqrx\" (UID: \"2664a881-44d7-470a-93d0-df1b3eaca0bc\") " pod="openshift-image-registry/node-ca-stqrx" Oct 30 17:11:42.712065 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.711045 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-multus-cni-dir\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.712065 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.711068 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-run-k8s-cni-cncf-io\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.712065 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.711092 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-systemd-units\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.712065 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.711120 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-run-ovn-kubernetes\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.712065 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.711147 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-cni-netd\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.712065 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.711150 2572 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 17:11:42.712065 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.711173 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f608efeb-1bda-4375-a428-c6db6341fd0b-ovn-node-metrics-cert\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.712065 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.711186 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/467187ee-7039-4bb4-adbe-38161ee7c336-hosts-file\") pod \"node-resolver-zgb2b\" (UID: \"467187ee-7039-4bb4-adbe-38161ee7c336\") " pod="openshift-dns/node-resolver-zgb2b" Oct 30 17:11:42.712065 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.711202 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2906e95e-3baa-4c79-ad9c-b2592c1d414f-kubelet-config\") pod \"global-pull-secret-syncer-5mtjr\" (UID: \"2906e95e-3baa-4c79-ad9c-b2592c1d414f\") " pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:11:42.712065 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.711249 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-cnibin\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.712065 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.711256 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2906e95e-3baa-4c79-ad9c-b2592c1d414f-kubelet-config\") pod \"global-pull-secret-syncer-5mtjr\" (UID: \"2906e95e-3baa-4c79-ad9c-b2592c1d414f\") " pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:11:42.712065 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.711280 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-etc-kubernetes\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.712065 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.711310 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9swv\" (UniqueName: \"kubernetes.io/projected/55f62602-c7b7-4a6c-b244-f6e48775733f-kube-api-access-m9swv\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.712065 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.711328 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-run-netns\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.712065 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.711344 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-registration-dir\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:11:42.712065 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.711363 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqrt8\" (UniqueName: \"kubernetes.io/projected/6178534d-9f75-49c7-a977-de65737cbcea-kube-api-access-nqrt8\") pod \"network-metrics-daemon-87jjv\" (UID: \"6178534d-9f75-49c7-a977-de65737cbcea\") " pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:11:42.712822 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.711379 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:11:42.712822 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.711396 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs2g4\" (UniqueName: \"kubernetes.io/projected/c7b7edd7-11cc-4fac-92ae-a08140056adb-kube-api-access-xs2g4\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:11:42.712822 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.711413 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-sysconfig\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.712822 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.711450 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd6ddeda-aec9-4872-9e43-084577cfcd7d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:11:42.712822 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.711468 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-system-cni-dir\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.712822 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.711493 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-multus-conf-dir\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.712822 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.711543 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-var-lib-openvswitch\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.712822 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.711569 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-kubernetes\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.712822 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.711593 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-run\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.712822 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.711615 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-cni-bin\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.712822 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.711638 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2906e95e-3baa-4c79-ad9c-b2592c1d414f-original-pull-secret\") pod \"global-pull-secret-syncer-5mtjr\" (UID: \"2906e95e-3baa-4c79-ad9c-b2592c1d414f\") " pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:11:42.712822 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.711660 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sh5q\" (UniqueName: \"kubernetes.io/projected/933be7f4-8d62-4735-807f-7aa1439583fa-kube-api-access-6sh5q\") pod \"iptables-alerter-pj5s7\" (UID: \"933be7f4-8d62-4735-807f-7aa1439583fa\") " pod="openshift-network-operator/iptables-alerter-pj5s7" Oct 30 17:11:42.712822 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.711685 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-slash\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.712822 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.711706 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f608efeb-1bda-4375-a428-c6db6341fd0b-ovnkube-config\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.712822 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.712210 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/55f62602-c7b7-4a6c-b244-f6e48775733f-multus-daemon-config\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.712822 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.712281 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-run-systemd\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.712822 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.712309 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f608efeb-1bda-4375-a428-c6db6341fd0b-ovnkube-config\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.713689 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.712515 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-host\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.713689 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.712784 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-multus-socket-dir-parent\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.713689 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.712820 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-hostroot\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.713689 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.712852 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.713689 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.712957 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2906e95e-3baa-4c79-ad9c-b2592c1d414f-dbus\") pod \"global-pull-secret-syncer-5mtjr\" (UID: \"2906e95e-3baa-4c79-ad9c-b2592c1d414f\") " pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:11:42.713689 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.712957 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd6ddeda-aec9-4872-9e43-084577cfcd7d-system-cni-dir\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:11:42.713689 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.713000 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-multus-cni-dir\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.713689 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.713022 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-device-dir\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:11:42.713689 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.710041 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-var-lib-cni-multus\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.713689 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.713028 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd6ddeda-aec9-4872-9e43-084577cfcd7d-cnibin\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:11:42.713689 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.713054 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-run-ovn-kubernetes\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.713689 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.713082 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-cni-netd\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.713689 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.713097 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-var-lib-kubelet\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.713689 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.713189 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-lib-modules\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.713689 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.713305 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-socket-dir\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:11:42.713689 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.713399 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-sysctl-d\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.713689 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.713440 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-registration-dir\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:11:42.714651 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.713488 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/933be7f4-8d62-4735-807f-7aa1439583fa-host-slash\") pod \"iptables-alerter-pj5s7\" (UID: \"933be7f4-8d62-4735-807f-7aa1439583fa\") " pod="openshift-network-operator/iptables-alerter-pj5s7" Oct 30 17:11:42.714651 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.713536 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/933be7f4-8d62-4735-807f-7aa1439583fa-iptables-alerter-script\") pod \"iptables-alerter-pj5s7\" (UID: \"933be7f4-8d62-4735-807f-7aa1439583fa\") " pod="openshift-network-operator/iptables-alerter-pj5s7" Oct 30 17:11:42.714651 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.713566 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-modprobe-d\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.714651 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.713576 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-systemd-units\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.714651 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.713763 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:11:42.714651 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.713919 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-sysconfig\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.714651 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.713971 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f608efeb-1bda-4375-a428-c6db6341fd0b-env-overrides\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.714651 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.713979 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd6ddeda-aec9-4872-9e43-084577cfcd7d-cni-binary-copy\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:11:42.714651 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.714549 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-system-cni-dir\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.714651 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.714610 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-multus-conf-dir\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.715194 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.714660 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-var-lib-openvswitch\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.715194 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.714713 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-kubernetes\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.715194 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.711568 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/467187ee-7039-4bb4-adbe-38161ee7c336-tmp-dir\") pod \"node-resolver-zgb2b\" (UID: \"467187ee-7039-4bb4-adbe-38161ee7c336\") " pod="openshift-dns/node-resolver-zgb2b" Oct 30 17:11:42.715194 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.714818 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-run\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.715194 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.714869 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-cni-bin\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.715194 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:42.714972 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:11:42.715194 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:42.715028 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2906e95e-3baa-4c79-ad9c-b2592c1d414f-original-pull-secret podName:2906e95e-3baa-4c79-ad9c-b2592c1d414f nodeName:}" failed. No retries permitted until 2026-10-30 17:11:43.215008815 +0000 UTC m=+3.139511349 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2906e95e-3baa-4c79-ad9c-b2592c1d414f-original-pull-secret") pod "global-pull-secret-syncer-5mtjr" (UID: "2906e95e-3baa-4c79-ad9c-b2592c1d414f") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:11:42.715622 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.715261 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-slash\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.715622 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.715311 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-log-socket\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.715622 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.715352 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-var-lib-kubelet\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.715622 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.715394 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-node-log\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.715763 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.715734 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-cnibin\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.715810 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.715784 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-etc-kubernetes\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.715868 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.715844 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2664a881-44d7-470a-93d0-df1b3eaca0bc-host\") pod \"node-ca-stqrx\" (UID: \"2664a881-44d7-470a-93d0-df1b3eaca0bc\") " pod="openshift-image-registry/node-ca-stqrx" Oct 30 17:11:42.715925 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.713028 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-run-k8s-cni-cncf-io\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.715925 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.715910 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-run-multus-certs\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.716060 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.716035 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-run-netns\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.716121 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.716104 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd6ddeda-aec9-4872-9e43-084577cfcd7d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:11:42.716361 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.716337 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-tuned\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.716468 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.716445 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f442e60f-8ccf-485e-9709-60a909d482bc-agent-certs\") pod \"konnectivity-agent-jnfks\" (UID: \"f442e60f-8ccf-485e-9709-60a909d482bc\") " pod="kube-system/konnectivity-agent-jnfks" Oct 30 17:11:42.716532 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.716472 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55f62602-c7b7-4a6c-b244-f6e48775733f-cni-binary-copy\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.716678 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.716653 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f608efeb-1bda-4375-a428-c6db6341fd0b-ovnkube-script-lib\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.716734 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.716679 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2664a881-44d7-470a-93d0-df1b3eaca0bc-serviceca\") pod \"node-ca-stqrx\" (UID: \"2664a881-44d7-470a-93d0-df1b3eaca0bc\") " pod="openshift-image-registry/node-ca-stqrx" Oct 30 17:11:42.717167 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.717107 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f442e60f-8ccf-485e-9709-60a909d482bc-konnectivity-ca\") pod \"konnectivity-agent-jnfks\" (UID: \"f442e60f-8ccf-485e-9709-60a909d482bc\") " pod="kube-system/konnectivity-agent-jnfks" Oct 30 17:11:42.717269 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.717202 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f608efeb-1bda-4375-a428-c6db6341fd0b-ovn-node-metrics-cert\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.717461 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.717444 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd6ddeda-aec9-4872-9e43-084577cfcd7d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:11:42.717587 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.717574 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d05e04a8-3ddf-4261-add0-c2324c6c9e21-tmp\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.734783 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.734757 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d49vx\" (UniqueName: \"kubernetes.io/projected/f608efeb-1bda-4375-a428-c6db6341fd0b-kube-api-access-d49vx\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.735632 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:42.735606 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:11:42.735742 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:42.735635 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:11:42.735742 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:42.735652 2572 projected.go:196] Error preparing data for projected volume kube-api-access-z4njs for pod openshift-network-diagnostics/network-check-target-t2pqm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:11:42.735851 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:42.735741 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d39ddd3a-781f-4084-b4e2-624b130c1025-kube-api-access-z4njs podName:d39ddd3a-781f-4084-b4e2-624b130c1025 nodeName:}" failed. No retries permitted until 2026-10-30 17:11:43.235723049 +0000 UTC m=+3.160225604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-z4njs" (UniqueName: "kubernetes.io/projected/d39ddd3a-781f-4084-b4e2-624b130c1025-kube-api-access-z4njs") pod "network-check-target-t2pqm" (UID: "d39ddd3a-781f-4084-b4e2-624b130c1025") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:11:42.737808 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.737709 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5mmm\" (UniqueName: \"kubernetes.io/projected/2664a881-44d7-470a-93d0-df1b3eaca0bc-kube-api-access-p5mmm\") pod \"node-ca-stqrx\" (UID: \"2664a881-44d7-470a-93d0-df1b3eaca0bc\") " pod="openshift-image-registry/node-ca-stqrx" Oct 30 17:11:42.740146 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.740118 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqrt8\" (UniqueName: \"kubernetes.io/projected/6178534d-9f75-49c7-a977-de65737cbcea-kube-api-access-nqrt8\") pod \"network-metrics-daemon-87jjv\" (UID: \"6178534d-9f75-49c7-a977-de65737cbcea\") " pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:11:42.742075 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.742054 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs2g4\" (UniqueName: \"kubernetes.io/projected/c7b7edd7-11cc-4fac-92ae-a08140056adb-kube-api-access-xs2g4\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:11:42.742075 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.742062 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9swv\" (UniqueName: \"kubernetes.io/projected/55f62602-c7b7-4a6c-b244-f6e48775733f-kube-api-access-m9swv\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.742243 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.742055 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r67qt\" (UniqueName: \"kubernetes.io/projected/cd6ddeda-aec9-4872-9e43-084577cfcd7d-kube-api-access-r67qt\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:11:42.742493 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.742472 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn8lc\" (UniqueName: \"kubernetes.io/projected/d05e04a8-3ddf-4261-add0-c2324c6c9e21-kube-api-access-jn8lc\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.743648 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.743622 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sh5q\" (UniqueName: \"kubernetes.io/projected/933be7f4-8d62-4735-807f-7aa1439583fa-kube-api-access-6sh5q\") pod \"iptables-alerter-pj5s7\" (UID: \"933be7f4-8d62-4735-807f-7aa1439583fa\") " pod="openshift-network-operator/iptables-alerter-pj5s7" Oct 30 17:11:42.744628 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.744399 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7n64\" (UniqueName: \"kubernetes.io/projected/467187ee-7039-4bb4-adbe-38161ee7c336-kube-api-access-p7n64\") pod \"node-resolver-zgb2b\" (UID: \"467187ee-7039-4bb4-adbe-38161ee7c336\") " pod="openshift-dns/node-resolver-zgb2b" Oct 30 17:11:42.836839 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.836798 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-stqrx" Oct 30 17:11:42.845107 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.845082 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jnfks" Oct 30 17:11:42.847255 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:42.847222 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2664a881_44d7_470a_93d0_df1b3eaca0bc.slice/crio-a41caba616cb22e3dc315b213de1e63cca96a70c659fb76d595aaa1273494dc0 WatchSource:0}: Error finding container a41caba616cb22e3dc315b213de1e63cca96a70c659fb76d595aaa1273494dc0: Status 404 returned error can't find the container with id a41caba616cb22e3dc315b213de1e63cca96a70c659fb76d595aaa1273494dc0 Oct 30 17:11:42.854444 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:42.854420 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf442e60f_8ccf_485e_9709_60a909d482bc.slice/crio-d76f4b44b8637a5ec189760f43d92a4b83070de536bf400ee08fdc21700fd49a WatchSource:0}: Error finding container d76f4b44b8637a5ec189760f43d92a4b83070de536bf400ee08fdc21700fd49a: Status 404 returned error can't find the container with id d76f4b44b8637a5ec189760f43d92a4b83070de536bf400ee08fdc21700fd49a Oct 30 17:11:42.859531 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.859506 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:11:42.865229 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.865202 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zgb2b" Oct 30 17:11:42.871885 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:42.871032 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd05e04a8_3ddf_4261_add0_c2324c6c9e21.slice/crio-44fcd866c23acce76db8d52203df5c385bfc735cbacc08b5f9d4a0e869f4f38c WatchSource:0}: Error finding container 44fcd866c23acce76db8d52203df5c385bfc735cbacc08b5f9d4a0e869f4f38c: Status 404 returned error can't find the container with id 44fcd866c23acce76db8d52203df5c385bfc735cbacc08b5f9d4a0e869f4f38c Oct 30 17:11:42.874767 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:42.874741 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod467187ee_7039_4bb4_adbe_38161ee7c336.slice/crio-1022106440a5e601a7e59d8d4ad406e94fd0f7fed1b15d885e20f8c70481c57e WatchSource:0}: Error finding container 1022106440a5e601a7e59d8d4ad406e94fd0f7fed1b15d885e20f8c70481c57e: Status 404 returned error can't find the container with id 1022106440a5e601a7e59d8d4ad406e94fd0f7fed1b15d885e20f8c70481c57e Oct 30 17:11:42.883046 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.883026 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:11:42.885277 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.885220 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-db6jk" Oct 30 17:11:42.894844 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.894825 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pj5s7" Oct 30 17:11:42.895300 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:42.895174 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd6ddeda_aec9_4872_9e43_084577cfcd7d.slice/crio-f585d4b8a2f2b3f0fe64149777c82cc42afa2e12c13f13b201f0d616827c6251 WatchSource:0}: Error finding container f585d4b8a2f2b3f0fe64149777c82cc42afa2e12c13f13b201f0d616827c6251: Status 404 returned error can't find the container with id f585d4b8a2f2b3f0fe64149777c82cc42afa2e12c13f13b201f0d616827c6251 Oct 30 17:11:42.896033 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:42.895976 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f62602_c7b7_4a6c_b244_f6e48775733f.slice/crio-8a9cf72d49e9c9eec7405103c22c662041354712ebc826ea78866070fbc7aff8 WatchSource:0}: Error finding container 8a9cf72d49e9c9eec7405103c22c662041354712ebc826ea78866070fbc7aff8: Status 404 returned error can't find the container with id 8a9cf72d49e9c9eec7405103c22c662041354712ebc826ea78866070fbc7aff8 Oct 30 17:11:42.906626 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:42.906601 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod933be7f4_8d62_4735_807f_7aa1439583fa.slice/crio-5cc65189d640b892290d7cf378248b8fbaeab9d260eac05bb6527647b0fed825 WatchSource:0}: Error finding container 5cc65189d640b892290d7cf378248b8fbaeab9d260eac05bb6527647b0fed825: Status 404 returned error can't find the container with id 5cc65189d640b892290d7cf378248b8fbaeab9d260eac05bb6527647b0fed825 Oct 30 17:11:42.920766 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.920741 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:11:42.923116 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.923094 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:11:42.931664 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:42.931627 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf608efeb_1bda_4375_a428_c6db6341fd0b.slice/crio-8956acc30b6dcb37c2361fa375236376b6d9aeb48ebe8359497860c622250a7a WatchSource:0}: Error finding container 8956acc30b6dcb37c2361fa375236376b6d9aeb48ebe8359497860c622250a7a: Status 404 returned error can't find the container with id 8956acc30b6dcb37c2361fa375236376b6d9aeb48ebe8359497860c622250a7a Oct 30 17:11:42.939539 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:11:42.939494 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b7edd7_11cc_4fac_92ae_a08140056adb.slice/crio-3f5fa8d664b221e5e8391e925d46ce9b0fa0c0e8061496e0a6ffb5aaad6138a7 WatchSource:0}: Error finding container 3f5fa8d664b221e5e8391e925d46ce9b0fa0c0e8061496e0a6ffb5aaad6138a7: Status 404 returned error can't find the container with id 3f5fa8d664b221e5e8391e925d46ce9b0fa0c0e8061496e0a6ffb5aaad6138a7 Oct 30 17:11:42.975455 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:42.975389 2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:11:43.218375 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:43.218312 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6178534d-9f75-49c7-a977-de65737cbcea-metrics-certs\") pod \"network-metrics-daemon-87jjv\" (UID: \"6178534d-9f75-49c7-a977-de65737cbcea\") " pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:11:43.218520 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:43.218417 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2906e95e-3baa-4c79-ad9c-b2592c1d414f-original-pull-secret\") pod \"global-pull-secret-syncer-5mtjr\" (UID: \"2906e95e-3baa-4c79-ad9c-b2592c1d414f\") " pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:11:43.218520 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:43.218476 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:11:43.218670 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:43.218531 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:11:43.218670 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:43.218545 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6178534d-9f75-49c7-a977-de65737cbcea-metrics-certs podName:6178534d-9f75-49c7-a977-de65737cbcea nodeName:}" failed. No retries permitted until 2026-10-30 17:11:44.218525334 +0000 UTC m=+4.143027887 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6178534d-9f75-49c7-a977-de65737cbcea-metrics-certs") pod "network-metrics-daemon-87jjv" (UID: "6178534d-9f75-49c7-a977-de65737cbcea") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:11:43.218670 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:43.218566 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2906e95e-3baa-4c79-ad9c-b2592c1d414f-original-pull-secret podName:2906e95e-3baa-4c79-ad9c-b2592c1d414f nodeName:}" failed. No retries permitted until 2026-10-30 17:11:44.218556478 +0000 UTC m=+4.143059011 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2906e95e-3baa-4c79-ad9c-b2592c1d414f-original-pull-secret") pod "global-pull-secret-syncer-5mtjr" (UID: "2906e95e-3baa-4c79-ad9c-b2592c1d414f") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:11:43.320791 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:43.320044 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4njs\" (UniqueName: \"kubernetes.io/projected/d39ddd3a-781f-4084-b4e2-624b130c1025-kube-api-access-z4njs\") pod \"network-check-target-t2pqm\" (UID: \"d39ddd3a-781f-4084-b4e2-624b130c1025\") " pod="openshift-network-diagnostics/network-check-target-t2pqm" Oct 30 17:11:43.320791 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:43.320254 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:11:43.320791 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:43.320274 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:11:43.320791 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:43.320288 2572 projected.go:196] Error preparing data for projected volume kube-api-access-z4njs for pod openshift-network-diagnostics/network-check-target-t2pqm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:11:43.320791 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:43.320376 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d39ddd3a-781f-4084-b4e2-624b130c1025-kube-api-access-z4njs podName:d39ddd3a-781f-4084-b4e2-624b130c1025 nodeName:}" failed. No retries permitted until 2026-10-30 17:11:44.320331635 +0000 UTC m=+4.244834186 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-z4njs" (UniqueName: "kubernetes.io/projected/d39ddd3a-781f-4084-b4e2-624b130c1025-kube-api-access-z4njs") pod "network-check-target-t2pqm" (UID: "d39ddd3a-781f-4084-b4e2-624b130c1025") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:11:43.561393 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:43.561256 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:06:43 +0000 UTC" deadline="2028-07-28 04:37:56.460785508 +0000 UTC" Oct 30 17:11:43.561393 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:43.561317 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15275h26m12.899472854s" Oct 30 17:11:43.623401 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:43.622862 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:11:43.623401 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:43.623016 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mtjr" podUID="2906e95e-3baa-4c79-ad9c-b2592c1d414f" Oct 30 17:11:43.665193 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:43.661707 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" event={"ID":"f608efeb-1bda-4375-a428-c6db6341fd0b","Type":"ContainerStarted","Data":"8956acc30b6dcb37c2361fa375236376b6d9aeb48ebe8359497860c622250a7a"} Oct 30 17:11:43.672796 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:43.672758 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-db6jk" event={"ID":"55f62602-c7b7-4a6c-b244-f6e48775733f","Type":"ContainerStarted","Data":"8a9cf72d49e9c9eec7405103c22c662041354712ebc826ea78866070fbc7aff8"} Oct 30 17:11:43.682357 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:43.682298 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zgb2b" event={"ID":"467187ee-7039-4bb4-adbe-38161ee7c336","Type":"ContainerStarted","Data":"1022106440a5e601a7e59d8d4ad406e94fd0f7fed1b15d885e20f8c70481c57e"} Oct 30 17:11:43.684859 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:43.684833 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-stqrx" event={"ID":"2664a881-44d7-470a-93d0-df1b3eaca0bc","Type":"ContainerStarted","Data":"a41caba616cb22e3dc315b213de1e63cca96a70c659fb76d595aaa1273494dc0"} Oct 30 17:11:43.696010 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:43.695915 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" event={"ID":"c7b7edd7-11cc-4fac-92ae-a08140056adb","Type":"ContainerStarted","Data":"3f5fa8d664b221e5e8391e925d46ce9b0fa0c0e8061496e0a6ffb5aaad6138a7"} Oct 30 17:11:43.717188 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:43.717088 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pj5s7" event={"ID":"933be7f4-8d62-4735-807f-7aa1439583fa","Type":"ContainerStarted","Data":"5cc65189d640b892290d7cf378248b8fbaeab9d260eac05bb6527647b0fed825"} Oct 30 17:11:43.735396 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:43.735351 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7gvrp" event={"ID":"cd6ddeda-aec9-4872-9e43-084577cfcd7d","Type":"ContainerStarted","Data":"f585d4b8a2f2b3f0fe64149777c82cc42afa2e12c13f13b201f0d616827c6251"} Oct 30 17:11:43.754006 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:43.749172 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" event={"ID":"d05e04a8-3ddf-4261-add0-c2324c6c9e21","Type":"ContainerStarted","Data":"44fcd866c23acce76db8d52203df5c385bfc735cbacc08b5f9d4a0e869f4f38c"} Oct 30 17:11:43.759682 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:43.757826 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jnfks" event={"ID":"f442e60f-8ccf-485e-9709-60a909d482bc","Type":"ContainerStarted","Data":"d76f4b44b8637a5ec189760f43d92a4b83070de536bf400ee08fdc21700fd49a"} Oct 30 17:11:44.226238 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:44.226119 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6178534d-9f75-49c7-a977-de65737cbcea-metrics-certs\") pod \"network-metrics-daemon-87jjv\" (UID: \"6178534d-9f75-49c7-a977-de65737cbcea\") " pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:11:44.226238 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:44.226199 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2906e95e-3baa-4c79-ad9c-b2592c1d414f-original-pull-secret\") pod \"global-pull-secret-syncer-5mtjr\" (UID: \"2906e95e-3baa-4c79-ad9c-b2592c1d414f\") " pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:11:44.226443 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:44.226335 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:11:44.226443 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:44.226398 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2906e95e-3baa-4c79-ad9c-b2592c1d414f-original-pull-secret podName:2906e95e-3baa-4c79-ad9c-b2592c1d414f nodeName:}" failed. No retries permitted until 2026-10-30 17:11:46.2263762 +0000 UTC m=+6.150878758 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2906e95e-3baa-4c79-ad9c-b2592c1d414f-original-pull-secret") pod "global-pull-secret-syncer-5mtjr" (UID: "2906e95e-3baa-4c79-ad9c-b2592c1d414f") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:11:44.226565 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:44.226471 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:11:44.226565 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:44.226509 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6178534d-9f75-49c7-a977-de65737cbcea-metrics-certs podName:6178534d-9f75-49c7-a977-de65737cbcea nodeName:}" failed. No retries permitted until 2026-10-30 17:11:46.226496118 +0000 UTC m=+6.150998668 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6178534d-9f75-49c7-a977-de65737cbcea-metrics-certs") pod "network-metrics-daemon-87jjv" (UID: "6178534d-9f75-49c7-a977-de65737cbcea") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:11:44.328822 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:44.328781 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4njs\" (UniqueName: \"kubernetes.io/projected/d39ddd3a-781f-4084-b4e2-624b130c1025-kube-api-access-z4njs\") pod \"network-check-target-t2pqm\" (UID: \"d39ddd3a-781f-4084-b4e2-624b130c1025\") " pod="openshift-network-diagnostics/network-check-target-t2pqm" Oct 30 17:11:44.329036 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:44.329017 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:11:44.329138 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:44.329053 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:11:44.329138 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:44.329067 2572 projected.go:196] Error preparing data for projected volume kube-api-access-z4njs for pod openshift-network-diagnostics/network-check-target-t2pqm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:11:44.329138 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:44.329128 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d39ddd3a-781f-4084-b4e2-624b130c1025-kube-api-access-z4njs podName:d39ddd3a-781f-4084-b4e2-624b130c1025 nodeName:}" failed. No retries permitted until 2026-10-30 17:11:46.329109465 +0000 UTC m=+6.253612013 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-z4njs" (UniqueName: "kubernetes.io/projected/d39ddd3a-781f-4084-b4e2-624b130c1025-kube-api-access-z4njs") pod "network-check-target-t2pqm" (UID: "d39ddd3a-781f-4084-b4e2-624b130c1025") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:11:44.561628 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:44.561529 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:06:43 +0000 UTC" deadline="2028-06-06 22:02:12.335550545 +0000 UTC" Oct 30 17:11:44.561628 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:44.561574 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14044h50m27.773981338s" Oct 30 17:11:44.624794 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:44.624528 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:11:44.624993 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:44.624915 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jjv" podUID="6178534d-9f75-49c7-a977-de65737cbcea" Oct 30 17:11:44.625488 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:44.625333 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t2pqm" Oct 30 17:11:44.625488 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:44.625422 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t2pqm" podUID="d39ddd3a-781f-4084-b4e2-624b130c1025" Oct 30 17:11:44.766079 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:44.766030 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-222.ec2.internal" event={"ID":"67161c7401d4577fb0911752be0668c2","Type":"ContainerStarted","Data":"fda2c96be0659a82de123de3859cddbf59249b57f4b433f9f21900d66be20c97"} Oct 30 17:11:44.776314 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:44.775697 2572 generic.go:356] "Generic (PLEG): container finished" podID="d961f94773b0889650472ea1a7f54825" containerID="1f14215c577069a9da506cd135905f95b124159c42614e72daf64985cf8c86c2" exitCode=0 Oct 30 17:11:44.776314 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:44.775747 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal" event={"ID":"d961f94773b0889650472ea1a7f54825","Type":"ContainerDied","Data":"1f14215c577069a9da506cd135905f95b124159c42614e72daf64985cf8c86c2"} Oct 30 17:11:44.854559 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:44.854495 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-222.ec2.internal" podStartSLOduration=2.854476329 podStartE2EDuration="2.854476329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:44.810090884 +0000 UTC m=+4.734593443" watchObservedRunningTime="2026-10-30 17:11:44.854476329 +0000 UTC m=+4.778978888" Oct 30 17:11:45.622276 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:45.622194 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:11:45.622753 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:45.622336 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mtjr" podUID="2906e95e-3baa-4c79-ad9c-b2592c1d414f" Oct 30 17:11:45.799522 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:45.799362 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal" event={"ID":"d961f94773b0889650472ea1a7f54825","Type":"ContainerStarted","Data":"61598f925a2bb36cc40a56cb98ec1a95972e30279ea51b60e459a50b5869b3be"} Oct 30 17:11:45.843136 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:45.842405 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal" podStartSLOduration=3.842392661 podStartE2EDuration="3.842392661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:45.841834645 +0000 UTC m=+5.766337203" watchObservedRunningTime="2026-10-30 17:11:45.842392661 +0000 UTC m=+5.766895218" Oct 30 17:11:46.243452 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:46.243412 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2906e95e-3baa-4c79-ad9c-b2592c1d414f-original-pull-secret\") pod \"global-pull-secret-syncer-5mtjr\" (UID: \"2906e95e-3baa-4c79-ad9c-b2592c1d414f\") " pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:11:46.243610 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:46.243499 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6178534d-9f75-49c7-a977-de65737cbcea-metrics-certs\") pod \"network-metrics-daemon-87jjv\" (UID: \"6178534d-9f75-49c7-a977-de65737cbcea\") " pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:11:46.243693 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:46.243617 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:11:46.243693 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:46.243644 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:11:46.243800 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:46.243695 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2906e95e-3baa-4c79-ad9c-b2592c1d414f-original-pull-secret podName:2906e95e-3baa-4c79-ad9c-b2592c1d414f nodeName:}" failed. No retries permitted until 2026-10-30 17:11:50.243673679 +0000 UTC m=+10.168176229 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2906e95e-3baa-4c79-ad9c-b2592c1d414f-original-pull-secret") pod "global-pull-secret-syncer-5mtjr" (UID: "2906e95e-3baa-4c79-ad9c-b2592c1d414f") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:11:46.243800 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:46.243716 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6178534d-9f75-49c7-a977-de65737cbcea-metrics-certs podName:6178534d-9f75-49c7-a977-de65737cbcea nodeName:}" failed. No retries permitted until 2026-10-30 17:11:50.243706189 +0000 UTC m=+10.168208730 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6178534d-9f75-49c7-a977-de65737cbcea-metrics-certs") pod "network-metrics-daemon-87jjv" (UID: "6178534d-9f75-49c7-a977-de65737cbcea") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:11:46.344675 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:46.344634 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4njs\" (UniqueName: \"kubernetes.io/projected/d39ddd3a-781f-4084-b4e2-624b130c1025-kube-api-access-z4njs\") pod \"network-check-target-t2pqm\" (UID: \"d39ddd3a-781f-4084-b4e2-624b130c1025\") " pod="openshift-network-diagnostics/network-check-target-t2pqm" Oct 30 17:11:46.344864 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:46.344830 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:11:46.344864 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:46.344852 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:11:46.344864 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:46.344865 2572 projected.go:196] Error preparing data for projected volume kube-api-access-z4njs for pod openshift-network-diagnostics/network-check-target-t2pqm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:11:46.345063 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:46.344947 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d39ddd3a-781f-4084-b4e2-624b130c1025-kube-api-access-z4njs podName:d39ddd3a-781f-4084-b4e2-624b130c1025 nodeName:}" failed. No retries permitted until 2026-10-30 17:11:50.34492797 +0000 UTC m=+10.269430508 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-z4njs" (UniqueName: "kubernetes.io/projected/d39ddd3a-781f-4084-b4e2-624b130c1025-kube-api-access-z4njs") pod "network-check-target-t2pqm" (UID: "d39ddd3a-781f-4084-b4e2-624b130c1025") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:11:46.625022 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:46.624987 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:11:46.625530 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:46.625140 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jjv" podUID="6178534d-9f75-49c7-a977-de65737cbcea" Oct 30 17:11:46.625530 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:46.625520 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t2pqm" Oct 30 17:11:46.625660 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:46.625616 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t2pqm" podUID="d39ddd3a-781f-4084-b4e2-624b130c1025" Oct 30 17:11:46.809323 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:46.806533 2572 generic.go:356] "Generic (PLEG): container finished" podID="cd6ddeda-aec9-4872-9e43-084577cfcd7d" containerID="1b53ed7e67d84f74e2642c1bb64495972105deaeb7c221a30edded531a51c575" exitCode=0 Oct 30 17:11:46.809323 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:46.807613 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7gvrp" event={"ID":"cd6ddeda-aec9-4872-9e43-084577cfcd7d","Type":"ContainerDied","Data":"1b53ed7e67d84f74e2642c1bb64495972105deaeb7c221a30edded531a51c575"} Oct 30 17:11:47.622981 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:47.622943 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:11:47.623192 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:47.623083 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mtjr" podUID="2906e95e-3baa-4c79-ad9c-b2592c1d414f" Oct 30 17:11:47.814835 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:47.814797 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jnfks" event={"ID":"f442e60f-8ccf-485e-9709-60a909d482bc","Type":"ContainerStarted","Data":"0388c4b7ce9270dce58775f0cf82f4b180d05cbb08513feab9291d7dab822feb"} Oct 30 17:11:48.623266 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:48.623223 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:11:48.623471 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:48.623370 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jjv" podUID="6178534d-9f75-49c7-a977-de65737cbcea" Oct 30 17:11:48.623788 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:48.623770 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t2pqm" Oct 30 17:11:48.623885 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:48.623857 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t2pqm" podUID="d39ddd3a-781f-4084-b4e2-624b130c1025" Oct 30 17:11:48.825289 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:48.825245 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-stqrx" event={"ID":"2664a881-44d7-470a-93d0-df1b3eaca0bc","Type":"ContainerStarted","Data":"4f4c978e01702ad332ca57bcad22e226f9ac1ed9c917ef32f66e301ec31c2dc8"} Oct 30 17:11:48.877533 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:48.877414 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jnfks" podStartSLOduration=3.937805939 podStartE2EDuration="7.877392629s" totalImagesPullingTime="3.93958669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:47.860801818 +0000 UTC m=+7.785304394" watchObservedRunningTime="2026-10-30 17:11:48.877392629 +0000 UTC m=+8.801895185" Oct 30 17:11:48.880728 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:48.878410 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-stqrx" podStartSLOduration=2.741539625 podStartE2EDuration="7.878390535s" totalImagesPullingTime="5.13685091s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:48.877230473 +0000 UTC m=+8.801733036" watchObservedRunningTime="2026-10-30 17:11:48.878390535 +0000 UTC m=+8.802893095" Oct 30 17:11:49.622803 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:49.622763 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:11:49.623003 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:49.622937 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mtjr" podUID="2906e95e-3baa-4c79-ad9c-b2592c1d414f" Oct 30 17:11:49.832312 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:49.832269 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" event={"ID":"c7b7edd7-11cc-4fac-92ae-a08140056adb","Type":"ContainerStarted","Data":"50d27d4bb4f2c8df03cdaa72e310770bfc752edbba7564111732fcda1d2de7f5"} Oct 30 17:11:49.863867 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:49.863833 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jnfks" Oct 30 17:11:49.864911 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:49.864885 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jnfks" Oct 30 17:11:50.281533 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:50.281484 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2906e95e-3baa-4c79-ad9c-b2592c1d414f-original-pull-secret\") pod \"global-pull-secret-syncer-5mtjr\" (UID: \"2906e95e-3baa-4c79-ad9c-b2592c1d414f\") " pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:11:50.281737 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:50.281596 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6178534d-9f75-49c7-a977-de65737cbcea-metrics-certs\") pod \"network-metrics-daemon-87jjv\" (UID: \"6178534d-9f75-49c7-a977-de65737cbcea\") " pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:11:50.281822 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:50.281738 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:11:50.281822 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:50.281813 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6178534d-9f75-49c7-a977-de65737cbcea-metrics-certs podName:6178534d-9f75-49c7-a977-de65737cbcea nodeName:}" failed. No retries permitted until 2026-10-30 17:11:58.28179279 +0000 UTC m=+18.206295340 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6178534d-9f75-49c7-a977-de65737cbcea-metrics-certs") pod "network-metrics-daemon-87jjv" (UID: "6178534d-9f75-49c7-a977-de65737cbcea") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:11:50.282290 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:50.282266 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:11:50.282382 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:50.282333 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2906e95e-3baa-4c79-ad9c-b2592c1d414f-original-pull-secret podName:2906e95e-3baa-4c79-ad9c-b2592c1d414f nodeName:}" failed. No retries permitted until 2026-10-30 17:11:58.282315642 +0000 UTC m=+18.206818191 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2906e95e-3baa-4c79-ad9c-b2592c1d414f-original-pull-secret") pod "global-pull-secret-syncer-5mtjr" (UID: "2906e95e-3baa-4c79-ad9c-b2592c1d414f") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:11:50.382127 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:50.382042 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4njs\" (UniqueName: \"kubernetes.io/projected/d39ddd3a-781f-4084-b4e2-624b130c1025-kube-api-access-z4njs\") pod \"network-check-target-t2pqm\" (UID: \"d39ddd3a-781f-4084-b4e2-624b130c1025\") " pod="openshift-network-diagnostics/network-check-target-t2pqm" Oct 30 17:11:50.382318 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:50.382217 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:11:50.382318 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:50.382248 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:11:50.382318 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:50.382262 2572 projected.go:196] Error preparing data for projected volume kube-api-access-z4njs for pod openshift-network-diagnostics/network-check-target-t2pqm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:11:50.382478 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:50.382327 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d39ddd3a-781f-4084-b4e2-624b130c1025-kube-api-access-z4njs podName:d39ddd3a-781f-4084-b4e2-624b130c1025 nodeName:}" failed. No retries permitted until 2026-10-30 17:11:58.382306515 +0000 UTC m=+18.306809065 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-z4njs" (UniqueName: "kubernetes.io/projected/d39ddd3a-781f-4084-b4e2-624b130c1025-kube-api-access-z4njs") pod "network-check-target-t2pqm" (UID: "d39ddd3a-781f-4084-b4e2-624b130c1025") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:11:50.626051 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:50.626013 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:11:50.626272 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:50.626184 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jjv" podUID="6178534d-9f75-49c7-a977-de65737cbcea" Oct 30 17:11:50.626631 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:50.626610 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t2pqm" Oct 30 17:11:50.626771 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:50.626706 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t2pqm" podUID="d39ddd3a-781f-4084-b4e2-624b130c1025" Oct 30 17:11:50.835278 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:50.835149 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jnfks" Oct 30 17:11:50.837365 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:50.837144 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jnfks" Oct 30 17:11:50.861458 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:50.861389 2572 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 17:11:51.549604 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:51.549497 2572 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T17:11:50.861406317Z","UUID":"5aded1d7-f30d-4531-beb0-829add3d643e","Handler":null,"Name":"","Endpoint":""} Oct 30 17:11:51.551463 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:51.551064 2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 17:11:51.551463 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:51.551093 2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 17:11:51.622878 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:51.622395 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:11:51.622878 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:51.622532 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mtjr" podUID="2906e95e-3baa-4c79-ad9c-b2592c1d414f" Oct 30 17:11:51.839090 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:51.838906 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-db6jk" event={"ID":"55f62602-c7b7-4a6c-b244-f6e48775733f","Type":"ContainerStarted","Data":"b3504cb7db59d9caacedda825586fdac857bdf85a53f8db092d46dd2bd7c1243"} Oct 30 17:11:51.841664 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:51.841240 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zgb2b" event={"ID":"467187ee-7039-4bb4-adbe-38161ee7c336","Type":"ContainerStarted","Data":"ee8ab3a0064735ee1db0735fdd2e80bf90151d098b3cbff1c3f539d8ec13171c"} Oct 30 17:11:51.845184 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:51.845154 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" event={"ID":"c7b7edd7-11cc-4fac-92ae-a08140056adb","Type":"ContainerStarted","Data":"f02788602e159fb2eb960a80eb69cd4d85700d242d940be30538d0698ca3f5f8"} Oct 30 17:11:51.848318 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:51.848288 2572 generic.go:356] "Generic (PLEG): container finished" podID="cd6ddeda-aec9-4872-9e43-084577cfcd7d" containerID="8fc8fb8996d9593ecdc982cbe18ca846f7c9c44f0bf544fef4e98a0701ffdfe2" exitCode=0 Oct 30 17:11:51.849239 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:51.849217 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7gvrp" event={"ID":"cd6ddeda-aec9-4872-9e43-084577cfcd7d","Type":"ContainerDied","Data":"8fc8fb8996d9593ecdc982cbe18ca846f7c9c44f0bf544fef4e98a0701ffdfe2"} Oct 30 17:11:51.927016 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:51.926965 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-db6jk" podStartSLOduration=2.70647836 podStartE2EDuration="10.92694861s" totalImagesPullingTime="8.22047025s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:51.884750744 +0000 UTC m=+11.809253305" watchObservedRunningTime="2026-10-30 17:11:51.92694861 +0000 UTC m=+11.851451164" Oct 30 17:11:51.974886 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:51.974819 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-zgb2b" podStartSLOduration=2.652018195 podStartE2EDuration="10.974798473s" totalImagesPullingTime="8.322780278s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:51.926445415 +0000 UTC m=+11.850947973" watchObservedRunningTime="2026-10-30 17:11:51.974798473 +0000 UTC m=+11.899301027" Oct 30 17:11:52.628414 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:52.628388 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t2pqm" Oct 30 17:11:52.628553 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:52.628510 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t2pqm" podUID="d39ddd3a-781f-4084-b4e2-624b130c1025" Oct 30 17:11:52.628615 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:52.628568 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:11:52.628687 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:52.628666 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jjv" podUID="6178534d-9f75-49c7-a977-de65737cbcea" Oct 30 17:11:52.854334 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:52.854302 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" event={"ID":"c7b7edd7-11cc-4fac-92ae-a08140056adb","Type":"ContainerStarted","Data":"884ed4c7662f715d708baa27e64af9d7e9b7de39540892cd55937b1baa832d1d"} Oct 30 17:11:52.856802 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:52.856775 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pj5s7" event={"ID":"933be7f4-8d62-4735-807f-7aa1439583fa","Type":"ContainerStarted","Data":"28bdc77021210748ead2d25dc08a7baf8272fe08bfa16e2977753f05b65a3eca"} Oct 30 17:11:52.859792 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:52.859771 2572 generic.go:356] "Generic (PLEG): container finished" podID="cd6ddeda-aec9-4872-9e43-084577cfcd7d" containerID="bd8c48abaf647b0718e65353956a31c298042887a627746b3a40f74dbc639590" exitCode=0 Oct 30 17:11:52.859981 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:52.859956 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7gvrp" event={"ID":"cd6ddeda-aec9-4872-9e43-084577cfcd7d","Type":"ContainerDied","Data":"bd8c48abaf647b0718e65353956a31c298042887a627746b3a40f74dbc639590"} Oct 30 17:11:52.897916 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:52.897844 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" podStartSLOduration=3.436449139 podStartE2EDuration="11.897830867s" totalImagesPullingTime="8.461381728s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:11:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:52.897667524 +0000 UTC m=+12.822170078" watchObservedRunningTime="2026-10-30 17:11:52.897830867 +0000 UTC m=+12.822333420" Oct 30 17:11:52.988189 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:52.988120 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pj5s7" podStartSLOduration=3.854370827 podStartE2EDuration="11.988100322s" totalImagesPullingTime="8.133729495s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:52.987844226 +0000 UTC m=+12.912346781" watchObservedRunningTime="2026-10-30 17:11:52.988100322 +0000 UTC m=+12.912602877" Oct 30 17:11:53.622628 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:53.622597 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:11:53.622746 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:53.622724 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mtjr" podUID="2906e95e-3baa-4c79-ad9c-b2592c1d414f" Oct 30 17:11:53.864986 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:53.864948 2572 generic.go:356] "Generic (PLEG): container finished" podID="cd6ddeda-aec9-4872-9e43-084577cfcd7d" containerID="2301b4325aa5093c02513414b0c9db48ebcf49537ce78679d820c41cd4d79c92" exitCode=0 Oct 30 17:11:53.865474 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:53.865005 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7gvrp" event={"ID":"cd6ddeda-aec9-4872-9e43-084577cfcd7d","Type":"ContainerDied","Data":"2301b4325aa5093c02513414b0c9db48ebcf49537ce78679d820c41cd4d79c92"} Oct 30 17:11:53.866679 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:53.866649 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" event={"ID":"d05e04a8-3ddf-4261-add0-c2324c6c9e21","Type":"ContainerStarted","Data":"4843a41da11899b43405f677657a954d82ab235a35e4ab491493c8dc4a92a558"} Oct 30 17:11:53.953129 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:53.953082 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" podStartSLOduration=2.260188585 podStartE2EDuration="12.953067375s" totalImagesPullingTime="10.69287879s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:53.952238235 +0000 UTC m=+13.876740790" watchObservedRunningTime="2026-10-30 17:11:53.953067375 +0000 UTC m=+13.877569928" Oct 30 17:11:54.622941 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:54.622902 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:11:54.622941 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:54.622931 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t2pqm" Oct 30 17:11:54.623180 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:54.623052 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jjv" podUID="6178534d-9f75-49c7-a977-de65737cbcea" Oct 30 17:11:54.623233 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:54.623177 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t2pqm" podUID="d39ddd3a-781f-4084-b4e2-624b130c1025" Oct 30 17:11:55.622662 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:55.622629 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:11:55.623382 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:55.622743 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mtjr" podUID="2906e95e-3baa-4c79-ad9c-b2592c1d414f" Oct 30 17:11:55.874630 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:55.874558 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7gvrp" event={"ID":"cd6ddeda-aec9-4872-9e43-084577cfcd7d","Type":"ContainerStarted","Data":"44e59dce5e2ddb6caf2141b8bf3e9dc8c70c93ca53ed78e9abbff5983e8acf5c"} Oct 30 17:11:56.625352 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:56.625320 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:11:56.625779 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:56.625450 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jjv" podUID="6178534d-9f75-49c7-a977-de65737cbcea" Oct 30 17:11:56.625779 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:56.625490 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t2pqm" Oct 30 17:11:56.625779 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:56.625597 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t2pqm" podUID="d39ddd3a-781f-4084-b4e2-624b130c1025" Oct 30 17:11:56.880297 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:56.880221 2572 generic.go:356] "Generic (PLEG): container finished" podID="cd6ddeda-aec9-4872-9e43-084577cfcd7d" containerID="44e59dce5e2ddb6caf2141b8bf3e9dc8c70c93ca53ed78e9abbff5983e8acf5c" exitCode=0 Oct 30 17:11:56.880297 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:56.880269 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7gvrp" event={"ID":"cd6ddeda-aec9-4872-9e43-084577cfcd7d","Type":"ContainerDied","Data":"44e59dce5e2ddb6caf2141b8bf3e9dc8c70c93ca53ed78e9abbff5983e8acf5c"} Oct 30 17:11:57.622432 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:57.622267 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:11:57.622637 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:57.622503 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mtjr" podUID="2906e95e-3baa-4c79-ad9c-b2592c1d414f" Oct 30 17:11:57.885223 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:57.885151 2572 generic.go:356] "Generic (PLEG): container finished" podID="cd6ddeda-aec9-4872-9e43-084577cfcd7d" containerID="3c3c9ee9b09b99e5d5170d9644f47e4a5bf6af86e22f4fb3ae61b125c57c0d2a" exitCode=0 Oct 30 17:11:57.885223 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:57.885200 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7gvrp" event={"ID":"cd6ddeda-aec9-4872-9e43-084577cfcd7d","Type":"ContainerDied","Data":"3c3c9ee9b09b99e5d5170d9644f47e4a5bf6af86e22f4fb3ae61b125c57c0d2a"} Oct 30 17:11:58.343601 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:58.343564 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2906e95e-3baa-4c79-ad9c-b2592c1d414f-original-pull-secret\") pod \"global-pull-secret-syncer-5mtjr\" (UID: \"2906e95e-3baa-4c79-ad9c-b2592c1d414f\") " pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:11:58.343743 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:58.343616 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6178534d-9f75-49c7-a977-de65737cbcea-metrics-certs\") pod \"network-metrics-daemon-87jjv\" (UID: \"6178534d-9f75-49c7-a977-de65737cbcea\") " pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:11:58.343743 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:58.343704 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:11:58.343807 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:58.343755 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6178534d-9f75-49c7-a977-de65737cbcea-metrics-certs podName:6178534d-9f75-49c7-a977-de65737cbcea nodeName:}" failed. No retries permitted until 2026-10-30 17:12:14.343742002 +0000 UTC m=+34.268244547 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6178534d-9f75-49c7-a977-de65737cbcea-metrics-certs") pod "network-metrics-daemon-87jjv" (UID: "6178534d-9f75-49c7-a977-de65737cbcea") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:11:58.343807 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:58.343704 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:11:58.343807 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:58.343787 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2906e95e-3baa-4c79-ad9c-b2592c1d414f-original-pull-secret podName:2906e95e-3baa-4c79-ad9c-b2592c1d414f nodeName:}" failed. No retries permitted until 2026-10-30 17:12:14.343781302 +0000 UTC m=+34.268283848 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2906e95e-3baa-4c79-ad9c-b2592c1d414f-original-pull-secret") pod "global-pull-secret-syncer-5mtjr" (UID: "2906e95e-3baa-4c79-ad9c-b2592c1d414f") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:11:58.443914 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:58.443837 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4njs\" (UniqueName: \"kubernetes.io/projected/d39ddd3a-781f-4084-b4e2-624b130c1025-kube-api-access-z4njs\") pod \"network-check-target-t2pqm\" (UID: \"d39ddd3a-781f-4084-b4e2-624b130c1025\") " pod="openshift-network-diagnostics/network-check-target-t2pqm" Oct 30 17:11:58.444038 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:58.443993 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:11:58.444038 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:58.444012 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:11:58.444038 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:58.444020 2572 projected.go:196] Error preparing data for projected volume kube-api-access-z4njs for pod openshift-network-diagnostics/network-check-target-t2pqm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:11:58.444126 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:58.444065 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d39ddd3a-781f-4084-b4e2-624b130c1025-kube-api-access-z4njs podName:d39ddd3a-781f-4084-b4e2-624b130c1025 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:14.444052388 +0000 UTC m=+34.368554921 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-z4njs" (UniqueName: "kubernetes.io/projected/d39ddd3a-781f-4084-b4e2-624b130c1025-kube-api-access-z4njs") pod "network-check-target-t2pqm" (UID: "d39ddd3a-781f-4084-b4e2-624b130c1025") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:11:58.622520 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:58.622349 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:11:58.622693 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:58.622374 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t2pqm" Oct 30 17:11:58.622693 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:58.622590 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jjv" podUID="6178534d-9f75-49c7-a977-de65737cbcea" Oct 30 17:11:58.622693 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:58.622653 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t2pqm" podUID="d39ddd3a-781f-4084-b4e2-624b130c1025" Oct 30 17:11:58.890114 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:58.890079 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7gvrp" event={"ID":"cd6ddeda-aec9-4872-9e43-084577cfcd7d","Type":"ContainerStarted","Data":"e87663f13f1ae107bf7503d0c81df937ca3d9e77bc38db874acdf44914a93ebf"} Oct 30 17:11:58.892887 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:58.892851 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" event={"ID":"f608efeb-1bda-4375-a428-c6db6341fd0b","Type":"ContainerStarted","Data":"326b284cec0d97bec6a3d0aa0451b158435d70a4afcfb05e3b33ad35abdcabda"} Oct 30 17:11:58.893021 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:58.892934 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" event={"ID":"f608efeb-1bda-4375-a428-c6db6341fd0b","Type":"ContainerStarted","Data":"eeba42dbb13dfe34667142e9e2c37c32c087e5afc05cb130bab52fb0067b8fc7"} Oct 30 17:11:58.893021 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:58.892979 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" event={"ID":"f608efeb-1bda-4375-a428-c6db6341fd0b","Type":"ContainerStarted","Data":"94f114f5fbc97644d03a51c5967367360f77a2021de53936a9a3c3f98585f54c"} Oct 30 17:11:58.893120 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:58.893023 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" event={"ID":"f608efeb-1bda-4375-a428-c6db6341fd0b","Type":"ContainerStarted","Data":"980a38184382760eaa27bd794763ae443e948c13358c27b2c57901dd26dbba13"} Oct 30 17:11:58.893120 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:58.893069 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" event={"ID":"f608efeb-1bda-4375-a428-c6db6341fd0b","Type":"ContainerStarted","Data":"78d31ba34a6e3cbc446518d6811004b228bff6423689137226cc84cd9034dc8a"} Oct 30 17:11:58.893202 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:58.893117 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" event={"ID":"f608efeb-1bda-4375-a428-c6db6341fd0b","Type":"ContainerStarted","Data":"60b7bedf5b78b7fefbaf334055e8d61233ae9e307c39878c81dad48b4428ab23"} Oct 30 17:11:58.936314 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:58.936277 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7gvrp" podStartSLOduration=6.975677867 podStartE2EDuration="17.936266115s" totalImagesPullingTime="10.960588248s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:11:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:58.936159955 +0000 UTC m=+18.860662508" watchObservedRunningTime="2026-10-30 17:11:58.936266115 +0000 UTC m=+18.860768669" Oct 30 17:11:59.622152 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:11:59.622118 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:11:59.622330 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:11:59.622224 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mtjr" podUID="2906e95e-3baa-4c79-ad9c-b2592c1d414f" Oct 30 17:12:00.623400 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:00.623370 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:12:00.623784 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:00.623502 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jjv" podUID="6178534d-9f75-49c7-a977-de65737cbcea" Oct 30 17:12:00.623784 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:00.623543 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t2pqm" Oct 30 17:12:00.623784 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:00.623599 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t2pqm" podUID="d39ddd3a-781f-4084-b4e2-624b130c1025" Oct 30 17:12:00.899646 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:00.899553 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" event={"ID":"f608efeb-1bda-4375-a428-c6db6341fd0b","Type":"ContainerStarted","Data":"431d1f330a446b99899468c159faadfe746df20fd4175e57621a5bae577cafca"} Oct 30 17:12:01.622042 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:01.622009 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:12:01.622220 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:01.622116 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mtjr" podUID="2906e95e-3baa-4c79-ad9c-b2592c1d414f" Oct 30 17:12:02.622947 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:02.622850 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t2pqm" Oct 30 17:12:02.623847 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:02.622855 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:12:02.623847 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:02.623003 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t2pqm" podUID="d39ddd3a-781f-4084-b4e2-624b130c1025" Oct 30 17:12:02.623847 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:02.623061 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jjv" podUID="6178534d-9f75-49c7-a977-de65737cbcea" Oct 30 17:12:02.907744 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:02.907663 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" event={"ID":"f608efeb-1bda-4375-a428-c6db6341fd0b","Type":"ContainerStarted","Data":"201b45c65d325af8d8e9beffb58bec298022e0465515e99b4e5984538a4b9517"} Oct 30 17:12:02.908051 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:02.908033 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:02.908131 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:02.908095 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:02.922215 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:02.922187 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:02.997815 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:02.997753 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" podStartSLOduration=7.010836965 podStartE2EDuration="21.997735733s" totalImagesPullingTime="14.986898768s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:02.996768799 +0000 UTC m=+22.921271354" watchObservedRunningTime="2026-10-30 17:12:02.997735733 +0000 UTC m=+22.922238290" Oct 30 17:12:03.623237 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:03.623202 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:12:03.623634 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:03.623329 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mtjr" podUID="2906e95e-3baa-4c79-ad9c-b2592c1d414f" Oct 30 17:12:03.910440 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:03.910366 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:03.924802 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:03.924776 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:04.435607 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:04.435579 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5mtjr"] Oct 30 17:12:04.435834 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:04.435676 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:12:04.435834 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:04.435753 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mtjr" podUID="2906e95e-3baa-4c79-ad9c-b2592c1d414f" Oct 30 17:12:04.438616 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:04.438592 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t2pqm"] Oct 30 17:12:04.438746 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:04.438690 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t2pqm" Oct 30 17:12:04.438790 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:04.438769 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t2pqm" podUID="d39ddd3a-781f-4084-b4e2-624b130c1025" Oct 30 17:12:04.442906 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:04.442885 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-87jjv"] Oct 30 17:12:04.443002 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:04.442974 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:12:04.443066 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:04.443042 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jjv" podUID="6178534d-9f75-49c7-a977-de65737cbcea" Oct 30 17:12:05.622832 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:05.622800 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:12:05.623290 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:05.622800 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t2pqm" Oct 30 17:12:05.623290 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:05.622917 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mtjr" podUID="2906e95e-3baa-4c79-ad9c-b2592c1d414f" Oct 30 17:12:05.623290 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:05.622989 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t2pqm" podUID="d39ddd3a-781f-4084-b4e2-624b130c1025" Oct 30 17:12:06.622324 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:06.622298 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:12:06.622462 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:06.622414 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jjv" podUID="6178534d-9f75-49c7-a977-de65737cbcea" Oct 30 17:12:07.623168 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:07.622993 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:12:07.623584 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:07.623030 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t2pqm" Oct 30 17:12:07.623584 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:07.623232 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mtjr" podUID="2906e95e-3baa-4c79-ad9c-b2592c1d414f" Oct 30 17:12:07.623584 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:07.623301 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t2pqm" podUID="d39ddd3a-781f-4084-b4e2-624b130c1025" Oct 30 17:12:08.622508 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:08.622476 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:12:08.622755 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:08.622610 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jjv" podUID="6178534d-9f75-49c7-a977-de65737cbcea" Oct 30 17:12:09.622766 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:09.622728 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:12:09.623224 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:09.622833 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mtjr" podUID="2906e95e-3baa-4c79-ad9c-b2592c1d414f" Oct 30 17:12:09.623224 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:09.622895 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t2pqm" Oct 30 17:12:09.623224 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:09.623014 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t2pqm" podUID="d39ddd3a-781f-4084-b4e2-624b130c1025" Oct 30 17:12:10.623472 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:10.623443 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:12:10.623822 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:10.623572 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87jjv" podUID="6178534d-9f75-49c7-a977-de65737cbcea" Oct 30 17:12:10.854849 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:10.854828 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-222.ec2.internal" event="NodeReady" Oct 30 17:12:10.855017 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:10.854991 2572 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 17:12:10.902148 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:10.902122 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-w5rnm"] Oct 30 17:12:10.957458 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:10.957432 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hwfbh"] Oct 30 17:12:10.957583 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:10.957551 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-w5rnm" Oct 30 17:12:10.968846 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:10.968828 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9jx9m"] Oct 30 17:12:10.968981 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:10.968961 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-hwfbh" Oct 30 17:12:10.977916 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:10.977895 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 17:12:10.978032 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:10.977973 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 17:12:10.991265 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:10.991240 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-mf9rd"] Oct 30 17:12:10.991388 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:10.991370 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-9jx9m" Oct 30 17:12:11.005929 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.005262 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t9hdn"] Oct 30 17:12:11.006326 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.006308 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" Oct 30 17:12:11.025293 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.025272 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 17:12:11.025480 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.025468 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-qjw5w\"" Oct 30 17:12:11.027507 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.027492 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-585b9dc6c5-dblkr"] Oct 30 17:12:11.027586 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.027572 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t9hdn" Oct 30 17:12:11.041316 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.041293 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txhrv\" (UniqueName: \"kubernetes.io/projected/03a0e7f9-9e86-4aa9-9864-205263f6a486-kube-api-access-txhrv\") pod \"kas-connection-checker-685895c7c9-w5rnm\" (UID: \"03a0e7f9-9e86-4aa9-9864-205263f6a486\") " pod="kube-system/kas-connection-checker-685895c7c9-w5rnm" Oct 30 17:12:11.041426 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.041342 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dz8c\" (UniqueName: \"kubernetes.io/projected/e30e6b59-b8e2-4125-ae7a-41edd1ce7389-kube-api-access-5dz8c\") pod \"kas-connection-checker-685895c7c9-hwfbh\" (UID: \"e30e6b59-b8e2-4125-ae7a-41edd1ce7389\") " pod="kube-system/kas-connection-checker-685895c7c9-hwfbh" Oct 30 17:12:11.051589 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.051570 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-kvpw6"] Oct 30 17:12:11.051688 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.051674 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:11.065648 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.065630 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-77579944f4-hctng"] Oct 30 17:12:11.065730 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.065703 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-kvpw6" Oct 30 17:12:11.076381 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.076363 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 17:12:11.076487 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.076437 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:12:11.076487 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.076441 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 17:12:11.076487 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.076441 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 17:12:11.076487 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.076479 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 17:12:11.076648 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.076518 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:12:11.085427 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.085400 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-ddb54c4f7-54tw2"] Oct 30 17:12:11.085534 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.085458 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:11.090132 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.090084 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 17:12:11.090132 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.090122 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 17:12:11.090312 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.090171 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 17:12:11.100045 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.100028 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-2tk8w"] Oct 30 17:12:11.100163 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.100149 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:11.114399 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.114377 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-6jbgg"] Oct 30 17:12:11.114518 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.114501 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2tk8w" Oct 30 17:12:11.126143 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.126098 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 17:12:11.126255 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.126123 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 17:12:11.126255 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.126148 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 17:12:11.126255 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.126193 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 17:12:11.126255 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.126216 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 17:12:11.126255 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.126228 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-8nbr8\"" Oct 30 17:12:11.126438 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.126286 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-599zc\"" Oct 30 17:12:11.126438 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.126315 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-786df\"" Oct 30 17:12:11.126438 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.126334 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 17:12:11.126438 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.126344 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 17:12:11.126438 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.126360 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 17:12:11.126438 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.126367 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-bvkn4\"" Oct 30 17:12:11.134435 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.134417 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-4w8fl"] Oct 30 17:12:11.134554 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.134538 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jbgg" Oct 30 17:12:11.141719 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.141697 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-tls\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:11.141814 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.141732 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txhrv\" (UniqueName: \"kubernetes.io/projected/03a0e7f9-9e86-4aa9-9864-205263f6a486-kube-api-access-txhrv\") pod \"kas-connection-checker-685895c7c9-w5rnm\" (UID: \"03a0e7f9-9e86-4aa9-9864-205263f6a486\") " pod="kube-system/kas-connection-checker-685895c7c9-w5rnm" Oct 30 17:12:11.141814 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.141759 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c2d3293d-1890-4fa2-b790-9f84622c9c8b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-kvpw6\" (UID: \"c2d3293d-1890-4fa2-b790-9f84622c9c8b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kvpw6" Oct 30 17:12:11.141814 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.141775 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/745f4f0f-74f7-4fff-9384-c8f45f601e33-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-t9hdn\" (UID: \"745f4f0f-74f7-4fff-9384-c8f45f601e33\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t9hdn" Oct 30 17:12:11.141954 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.141925 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1eab122e-07da-4307-b7f2-914cbe2b4237-config\") pod \"console-operator-7cd56b8dbc-mf9rd\" (UID: \"1eab122e-07da-4307-b7f2-914cbe2b4237\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" Oct 30 17:12:11.142017 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.141994 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dz8c\" (UniqueName: \"kubernetes.io/projected/e30e6b59-b8e2-4125-ae7a-41edd1ce7389-kube-api-access-5dz8c\") pod \"kas-connection-checker-685895c7c9-hwfbh\" (UID: \"e30e6b59-b8e2-4125-ae7a-41edd1ce7389\") " pod="kube-system/kas-connection-checker-685895c7c9-hwfbh" Oct 30 17:12:11.142057 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.142042 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kcdc\" (UniqueName: \"kubernetes.io/projected/c7ac81c7-e1f0-4549-886d-d7b8b893b614-kube-api-access-5kcdc\") pod \"kas-connection-checker-685895c7c9-9jx9m\" (UID: \"c7ac81c7-e1f0-4549-886d-d7b8b893b614\") " pod="kube-system/kas-connection-checker-685895c7c9-9jx9m" Oct 30 17:12:11.142093 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.142071 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1eab122e-07da-4307-b7f2-914cbe2b4237-serving-cert\") pod \"console-operator-7cd56b8dbc-mf9rd\" (UID: \"1eab122e-07da-4307-b7f2-914cbe2b4237\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" Oct 30 17:12:11.142142 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.142102 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-bound-sa-token\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:11.142202 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.142170 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-certificates\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:11.142202 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.142198 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brgv8\" (UniqueName: \"kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-kube-api-access-brgv8\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:11.142298 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.142219 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c2d3293d-1890-4fa2-b790-9f84622c9c8b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kvpw6\" (UID: \"c2d3293d-1890-4fa2-b790-9f84622c9c8b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kvpw6" Oct 30 17:12:11.142298 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.142253 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/745f4f0f-74f7-4fff-9384-c8f45f601e33-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-t9hdn\" (UID: \"745f4f0f-74f7-4fff-9384-c8f45f601e33\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t9hdn" Oct 30 17:12:11.142410 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.142383 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-trusted-ca\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:11.142464 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.142428 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-ca-trust-extracted\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:11.142464 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.142456 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9kmm\" (UniqueName: \"kubernetes.io/projected/1eab122e-07da-4307-b7f2-914cbe2b4237-kube-api-access-f9kmm\") pod \"console-operator-7cd56b8dbc-mf9rd\" (UID: \"1eab122e-07da-4307-b7f2-914cbe2b4237\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" Oct 30 17:12:11.142535 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.142495 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-image-registry-private-configuration\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:11.142535 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.142525 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-installation-pull-secrets\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:11.142633 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.142604 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztbbl\" (UniqueName: \"kubernetes.io/projected/745f4f0f-74f7-4fff-9384-c8f45f601e33-kube-api-access-ztbbl\") pod \"cluster-samples-operator-6ddb6c8b94-t9hdn\" (UID: \"745f4f0f-74f7-4fff-9384-c8f45f601e33\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t9hdn" Oct 30 17:12:11.142633 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.142620 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1eab122e-07da-4307-b7f2-914cbe2b4237-trusted-ca\") pod \"console-operator-7cd56b8dbc-mf9rd\" (UID: \"1eab122e-07da-4307-b7f2-914cbe2b4237\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" Oct 30 17:12:11.160802 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.160786 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-4w8fl" Oct 30 17:12:11.160947 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.160732 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4x66d"] Oct 30 17:12:11.165831 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.165813 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txhrv\" (UniqueName: \"kubernetes.io/projected/03a0e7f9-9e86-4aa9-9864-205263f6a486-kube-api-access-txhrv\") pod \"kas-connection-checker-685895c7c9-w5rnm\" (UID: \"03a0e7f9-9e86-4aa9-9864-205263f6a486\") " pod="kube-system/kas-connection-checker-685895c7c9-w5rnm" Oct 30 17:12:11.165914 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.165854 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dz8c\" (UniqueName: \"kubernetes.io/projected/e30e6b59-b8e2-4125-ae7a-41edd1ce7389-kube-api-access-5dz8c\") pod \"kas-connection-checker-685895c7c9-hwfbh\" (UID: \"e30e6b59-b8e2-4125-ae7a-41edd1ce7389\") " pod="kube-system/kas-connection-checker-685895c7c9-hwfbh" Oct 30 17:12:11.171086 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.171062 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-z6b8h"] Oct 30 17:12:11.171303 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.171167 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4x66d" Oct 30 17:12:11.177163 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.177141 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 17:12:11.177255 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.177228 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 17:12:11.177255 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.177240 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 17:12:11.177360 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.177255 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 17:12:11.177360 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.177333 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 17:12:11.177360 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.177342 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 17:12:11.177507 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.177369 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 17:12:11.177507 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.177378 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 17:12:11.177507 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.177378 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:12:11.177507 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.177439 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 17:12:11.177686 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.177571 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 17:12:11.178288 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.178118 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 17:12:11.178551 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.178535 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 17:12:11.179294 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.179278 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jccch"] Oct 30 17:12:11.179413 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.179398 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z6b8h" Oct 30 17:12:11.190930 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.190914 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:12:11.191044 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.191033 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 17:12:11.193633 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.193614 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 17:12:11.193633 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.193628 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 17:12:11.203121 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.203103 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hwfbh"] Oct 30 17:12:11.203174 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.203130 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-w5rnm"] Oct 30 17:12:11.203174 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.203144 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-22vlj"] Oct 30 17:12:11.203235 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.203213 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jccch" Oct 30 17:12:11.225923 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.225902 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t9hdn"] Oct 30 17:12:11.226011 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.225932 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rdjn5"] Oct 30 17:12:11.226011 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.225934 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-22vlj" Oct 30 17:12:11.226604 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.226589 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 17:12:11.226955 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.226933 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 17:12:11.226955 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.226943 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 17:12:11.227082 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.226962 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-j8vzc\"" Oct 30 17:12:11.227082 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.226996 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 17:12:11.227082 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.227009 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-r5hq7\"" Oct 30 17:12:11.227082 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.227020 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-xfcrk\"" Oct 30 17:12:11.227351 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.227335 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hr7mp\"" Oct 30 17:12:11.227413 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.227335 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-7qw5s\"" Oct 30 17:12:11.227413 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.227362 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-5pzbw\"" Oct 30 17:12:11.234870 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.234842 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-mf9rd"] Oct 30 17:12:11.234945 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.234886 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9jx9m"] Oct 30 17:12:11.234945 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.234907 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-kvpw6"] Oct 30 17:12:11.234945 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.234927 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-6jbgg"] Oct 30 17:12:11.234945 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.234936 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rdjn5" Oct 30 17:12:11.234945 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.234946 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-585b9dc6c5-dblkr"] Oct 30 17:12:11.235166 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.234967 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77579944f4-hctng"] Oct 30 17:12:11.235166 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.234989 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-2tk8w"] Oct 30 17:12:11.235166 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.235008 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-22vlj"] Oct 30 17:12:11.235166 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.235033 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rdjn5"] Oct 30 17:12:11.235166 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.235050 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-z6b8h"] Oct 30 17:12:11.235166 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.235070 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4x66d"] Oct 30 17:12:11.235166 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.235090 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jccch"] Oct 30 17:12:11.235166 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.235116 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-4w8fl"] Oct 30 17:12:11.235166 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.235137 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-ddb54c4f7-54tw2"] Oct 30 17:12:11.240198 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.240177 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rz5ks\"" Oct 30 17:12:11.240296 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.240262 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 17:12:11.243227 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.243207 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-image-registry-private-configuration\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:11.243307 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.243240 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-tls\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:11.243307 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.243268 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztbbl\" (UniqueName: \"kubernetes.io/projected/745f4f0f-74f7-4fff-9384-c8f45f601e33-kube-api-access-ztbbl\") pod \"cluster-samples-operator-6ddb6c8b94-t9hdn\" (UID: \"745f4f0f-74f7-4fff-9384-c8f45f601e33\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t9hdn" Oct 30 17:12:11.243419 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.243404 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-tls\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:11.243476 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.243434 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c2d3293d-1890-4fa2-b790-9f84622c9c8b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-kvpw6\" (UID: \"c2d3293d-1890-4fa2-b790-9f84622c9c8b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kvpw6" Oct 30 17:12:11.243476 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.243465 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rttdq\" (UniqueName: \"kubernetes.io/projected/cc71466d-eac0-460b-af50-3204f07b3f35-kube-api-access-rttdq\") pod \"cluster-monitoring-operator-884576c57-6jbgg\" (UID: \"cc71466d-eac0-460b-af50-3204f07b3f35\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jbgg" Oct 30 17:12:11.243573 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.243496 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1eab122e-07da-4307-b7f2-914cbe2b4237-config\") pod \"console-operator-7cd56b8dbc-mf9rd\" (UID: \"1eab122e-07da-4307-b7f2-914cbe2b4237\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" Oct 30 17:12:11.243573 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.243529 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:12:11.243573 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.243545 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-585b9dc6c5-dblkr: secret "image-registry-tls" not found Oct 30 17:12:11.243573 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.243551 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csqbr\" (UniqueName: \"kubernetes.io/projected/2989c0bb-0ed6-41b2-ad3e-bb6e85bd60b9-kube-api-access-csqbr\") pod \"volume-data-source-validator-58c9f5f46d-4x66d\" (UID: \"2989c0bb-0ed6-41b2-ad3e-bb6e85bd60b9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4x66d" Oct 30 17:12:11.243573 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.243574 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-metrics-certs\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:11.243843 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.243592 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-tls podName:b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce nodeName:}" failed. No retries permitted until 2026-10-30 17:12:11.743574994 +0000 UTC m=+31.668077526 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-tls") pod "image-registry-585b9dc6c5-dblkr" (UID: "b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce") : secret "image-registry-tls" not found Oct 30 17:12:11.243843 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.243624 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/16cd937e-4c6d-45d0-a9de-f77dc836d082-snapshots\") pod \"insights-operator-7c4986d449-4w8fl\" (UID: \"16cd937e-4c6d-45d0-a9de-f77dc836d082\") " pod="openshift-insights/insights-operator-7c4986d449-4w8fl" Oct 30 17:12:11.243843 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.243657 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-certificates\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:11.243843 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.243682 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-certificates\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:11.243843 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.243711 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd74ba94-86d0-4707-94e5-6822c4ab5751-trusted-ca\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:11.243843 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.243735 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx4rg\" (UniqueName: \"kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-kube-api-access-qx4rg\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:11.243843 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.243758 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16cd937e-4c6d-45d0-a9de-f77dc836d082-service-ca-bundle\") pod \"insights-operator-7c4986d449-4w8fl\" (UID: \"16cd937e-4c6d-45d0-a9de-f77dc836d082\") " pod="openshift-insights/insights-operator-7c4986d449-4w8fl" Oct 30 17:12:11.243843 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.243787 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtlv4\" (UniqueName: \"kubernetes.io/projected/16cd937e-4c6d-45d0-a9de-f77dc836d082-kube-api-access-wtlv4\") pod \"insights-operator-7c4986d449-4w8fl\" (UID: \"16cd937e-4c6d-45d0-a9de-f77dc836d082\") " pod="openshift-insights/insights-operator-7c4986d449-4w8fl" Oct 30 17:12:11.243843 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.243822 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-trusted-ca\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:11.244341 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.243855 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9kmm\" (UniqueName: \"kubernetes.io/projected/1eab122e-07da-4307-b7f2-914cbe2b4237-kube-api-access-f9kmm\") pod \"console-operator-7cd56b8dbc-mf9rd\" (UID: \"1eab122e-07da-4307-b7f2-914cbe2b4237\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" Oct 30 17:12:11.244341 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.243986 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc71466d-eac0-460b-af50-3204f07b3f35-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6jbgg\" (UID: \"cc71466d-eac0-460b-af50-3204f07b3f35\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jbgg" Oct 30 17:12:11.244341 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.244235 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-ca-trust-extracted\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:11.244341 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.244275 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-certificates\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:11.244341 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.244274 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1eab122e-07da-4307-b7f2-914cbe2b4237-config\") pod \"console-operator-7cd56b8dbc-mf9rd\" (UID: \"1eab122e-07da-4307-b7f2-914cbe2b4237\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" Oct 30 17:12:11.244341 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.244299 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd74ba94-86d0-4707-94e5-6822c4ab5751-installation-pull-secrets\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:11.244341 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.244338 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cc71466d-eac0-460b-af50-3204f07b3f35-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6jbgg\" (UID: \"cc71466d-eac0-460b-af50-3204f07b3f35\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jbgg" Oct 30 17:12:11.244341 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.244340 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c2d3293d-1890-4fa2-b790-9f84622c9c8b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-kvpw6\" (UID: \"c2d3293d-1890-4fa2-b790-9f84622c9c8b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kvpw6" Oct 30 17:12:11.244681 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.244385 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-installation-pull-secrets\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:11.244681 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.244409 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16cd937e-4c6d-45d0-a9de-f77dc836d082-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4w8fl\" (UID: \"16cd937e-4c6d-45d0-a9de-f77dc836d082\") " pod="openshift-insights/insights-operator-7c4986d449-4w8fl" Oct 30 17:12:11.244681 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.244420 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-trusted-ca\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:11.244681 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.244533 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-ca-trust-extracted\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:11.244681 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.244540 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1eab122e-07da-4307-b7f2-914cbe2b4237-trusted-ca\") pod \"console-operator-7cd56b8dbc-mf9rd\" (UID: \"1eab122e-07da-4307-b7f2-914cbe2b4237\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" Oct 30 17:12:11.244681 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.244580 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-default-certificate\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:11.244681 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.244610 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/16cd937e-4c6d-45d0-a9de-f77dc836d082-serving-cert\") pod \"insights-operator-7c4986d449-4w8fl\" (UID: \"16cd937e-4c6d-45d0-a9de-f77dc836d082\") " pod="openshift-insights/insights-operator-7c4986d449-4w8fl" Oct 30 17:12:11.244681 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.244665 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/745f4f0f-74f7-4fff-9384-c8f45f601e33-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-t9hdn\" (UID: \"745f4f0f-74f7-4fff-9384-c8f45f601e33\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t9hdn" Oct 30 17:12:11.245029 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.244691 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/924ac066-ff98-4320-9eb2-268f0a39ca33-service-ca-bundle\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:11.245029 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.244715 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/195f952d-ee96-485d-99aa-8b47e6633c97-serving-cert\") pod \"service-ca-operator-544d749c8c-2tk8w\" (UID: \"195f952d-ee96-485d-99aa-8b47e6633c97\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2tk8w" Oct 30 17:12:11.245029 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.244742 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrqrz\" (UniqueName: \"kubernetes.io/projected/924ac066-ff98-4320-9eb2-268f0a39ca33-kube-api-access-mrqrz\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:11.245029 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.244754 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 17:12:11.245029 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.244767 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l25lh\" (UniqueName: \"kubernetes.io/projected/195f952d-ee96-485d-99aa-8b47e6633c97-kube-api-access-l25lh\") pod \"service-ca-operator-544d749c8c-2tk8w\" (UID: \"195f952d-ee96-485d-99aa-8b47e6633c97\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2tk8w" Oct 30 17:12:11.245029 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.244814 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/745f4f0f-74f7-4fff-9384-c8f45f601e33-samples-operator-tls podName:745f4f0f-74f7-4fff-9384-c8f45f601e33 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:11.744798172 +0000 UTC m=+31.669300719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/745f4f0f-74f7-4fff-9384-c8f45f601e33-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-t9hdn" (UID: "745f4f0f-74f7-4fff-9384-c8f45f601e33") : secret "samples-operator-tls" not found Oct 30 17:12:11.245029 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.244882 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kcdc\" (UniqueName: \"kubernetes.io/projected/c7ac81c7-e1f0-4549-886d-d7b8b893b614-kube-api-access-5kcdc\") pod \"kas-connection-checker-685895c7c9-9jx9m\" (UID: \"c7ac81c7-e1f0-4549-886d-d7b8b893b614\") " pod="kube-system/kas-connection-checker-685895c7c9-9jx9m" Oct 30 17:12:11.245029 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.244914 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1eab122e-07da-4307-b7f2-914cbe2b4237-serving-cert\") pod \"console-operator-7cd56b8dbc-mf9rd\" (UID: \"1eab122e-07da-4307-b7f2-914cbe2b4237\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" Oct 30 17:12:11.245029 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.244943 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd74ba94-86d0-4707-94e5-6822c4ab5751-ca-trust-extracted\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:11.245029 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.244990 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-bound-sa-token\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:11.245029 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.245027 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-bound-sa-token\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:11.245559 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.245054 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd74ba94-86d0-4707-94e5-6822c4ab5751-image-registry-private-configuration\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:11.245559 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.245085 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brgv8\" (UniqueName: \"kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-kube-api-access-brgv8\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:11.245559 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.245113 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c2d3293d-1890-4fa2-b790-9f84622c9c8b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kvpw6\" (UID: \"c2d3293d-1890-4fa2-b790-9f84622c9c8b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kvpw6" Oct 30 17:12:11.245559 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.245138 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/745f4f0f-74f7-4fff-9384-c8f45f601e33-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-t9hdn\" (UID: \"745f4f0f-74f7-4fff-9384-c8f45f601e33\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t9hdn" Oct 30 17:12:11.245559 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.245170 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-stats-auth\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:11.245559 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.245187 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1eab122e-07da-4307-b7f2-914cbe2b4237-trusted-ca\") pod \"console-operator-7cd56b8dbc-mf9rd\" (UID: \"1eab122e-07da-4307-b7f2-914cbe2b4237\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" Oct 30 17:12:11.245559 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.245205 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/195f952d-ee96-485d-99aa-8b47e6633c97-config\") pod \"service-ca-operator-544d749c8c-2tk8w\" (UID: \"195f952d-ee96-485d-99aa-8b47e6633c97\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2tk8w" Oct 30 17:12:11.245559 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.245308 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16cd937e-4c6d-45d0-a9de-f77dc836d082-tmp\") pod \"insights-operator-7c4986d449-4w8fl\" (UID: \"16cd937e-4c6d-45d0-a9de-f77dc836d082\") " pod="openshift-insights/insights-operator-7c4986d449-4w8fl" Oct 30 17:12:11.245559 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.245317 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 17:12:11.245559 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.245436 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2d3293d-1890-4fa2-b790-9f84622c9c8b-networking-console-plugin-cert podName:c2d3293d-1890-4fa2-b790-9f84622c9c8b nodeName:}" failed. No retries permitted until 2026-10-30 17:12:11.745420963 +0000 UTC m=+31.669923497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c2d3293d-1890-4fa2-b790-9f84622c9c8b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kvpw6" (UID: "c2d3293d-1890-4fa2-b790-9f84622c9c8b") : secret "networking-console-plugin-cert" not found Oct 30 17:12:11.246049 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.245941 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-image-registry-private-configuration\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:11.246049 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.245949 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/745f4f0f-74f7-4fff-9384-c8f45f601e33-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-t9hdn\" (UID: \"745f4f0f-74f7-4fff-9384-c8f45f601e33\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t9hdn" Oct 30 17:12:11.246667 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.246645 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-installation-pull-secrets\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:11.246999 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.246983 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1eab122e-07da-4307-b7f2-914cbe2b4237-serving-cert\") pod \"console-operator-7cd56b8dbc-mf9rd\" (UID: \"1eab122e-07da-4307-b7f2-914cbe2b4237\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" Oct 30 17:12:11.249442 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.249426 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 17:12:11.249682 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.249652 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zkp55\"" Oct 30 17:12:11.269160 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.268926 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztbbl\" (UniqueName: \"kubernetes.io/projected/745f4f0f-74f7-4fff-9384-c8f45f601e33-kube-api-access-ztbbl\") pod \"cluster-samples-operator-6ddb6c8b94-t9hdn\" (UID: \"745f4f0f-74f7-4fff-9384-c8f45f601e33\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t9hdn" Oct 30 17:12:11.269715 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.269687 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-bound-sa-token\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:11.269813 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.269768 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kcdc\" (UniqueName: \"kubernetes.io/projected/c7ac81c7-e1f0-4549-886d-d7b8b893b614-kube-api-access-5kcdc\") pod \"kas-connection-checker-685895c7c9-9jx9m\" (UID: \"c7ac81c7-e1f0-4549-886d-d7b8b893b614\") " pod="kube-system/kas-connection-checker-685895c7c9-9jx9m" Oct 30 17:12:11.270120 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.270099 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brgv8\" (UniqueName: \"kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-kube-api-access-brgv8\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:11.270165 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.270131 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9kmm\" (UniqueName: \"kubernetes.io/projected/1eab122e-07da-4307-b7f2-914cbe2b4237-kube-api-access-f9kmm\") pod \"console-operator-7cd56b8dbc-mf9rd\" (UID: \"1eab122e-07da-4307-b7f2-914cbe2b4237\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" Oct 30 17:12:11.277832 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.277812 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-w5rnm" Oct 30 17:12:11.277931 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.277840 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 17:12:11.277931 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.277882 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 17:12:11.277931 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.277901 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:12:11.278040 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.277928 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 17:12:11.278087 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.278074 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 17:12:11.278121 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.278102 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 17:12:11.299581 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.299563 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-hwfbh" Oct 30 17:12:11.318214 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.318196 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-9jx9m" Oct 30 17:12:11.342028 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.342001 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" Oct 30 17:12:11.346288 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.346263 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-tls\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:11.346390 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.346333 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4qm2\" (UniqueName: \"kubernetes.io/projected/27ac623c-2f57-44ec-a1b1-3ad87909452d-kube-api-access-g4qm2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jccch\" (UID: \"27ac623c-2f57-44ec-a1b1-3ad87909452d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jccch" Oct 30 17:12:11.346390 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.346361 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2939b3a2-1950-44c9-802c-a53861aa7aa8-metrics-tls\") pod \"dns-default-22vlj\" (UID: \"2939b3a2-1950-44c9-802c-a53861aa7aa8\") " pod="openshift-dns/dns-default-22vlj" Oct 30 17:12:11.346390 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.346376 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:12:11.346390 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.346383 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffnbr\" (UniqueName: \"kubernetes.io/projected/2939b3a2-1950-44c9-802c-a53861aa7aa8-kube-api-access-ffnbr\") pod \"dns-default-22vlj\" (UID: \"2939b3a2-1950-44c9-802c-a53861aa7aa8\") " pod="openshift-dns/dns-default-22vlj" Oct 30 17:12:11.346390 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.346393 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77579944f4-hctng: secret "image-registry-tls" not found Oct 30 17:12:11.346610 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.346412 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rttdq\" (UniqueName: \"kubernetes.io/projected/cc71466d-eac0-460b-af50-3204f07b3f35-kube-api-access-rttdq\") pod \"cluster-monitoring-operator-884576c57-6jbgg\" (UID: \"cc71466d-eac0-460b-af50-3204f07b3f35\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jbgg" Oct 30 17:12:11.346610 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.346439 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-tls podName:fd74ba94-86d0-4707-94e5-6822c4ab5751 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:11.846423576 +0000 UTC m=+31.770926108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-tls") pod "image-registry-77579944f4-hctng" (UID: "fd74ba94-86d0-4707-94e5-6822c4ab5751") : secret "image-registry-tls" not found Oct 30 17:12:11.346610 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.346466 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2939b3a2-1950-44c9-802c-a53861aa7aa8-tmp-dir\") pod \"dns-default-22vlj\" (UID: \"2939b3a2-1950-44c9-802c-a53861aa7aa8\") " pod="openshift-dns/dns-default-22vlj" Oct 30 17:12:11.346610 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.346487 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zz48\" (UniqueName: \"kubernetes.io/projected/e5f38da3-fe1f-4f36-8ed3-12cb01bf128b-kube-api-access-5zz48\") pod \"network-check-source-6f78d59f99-z6b8h\" (UID: \"e5f38da3-fe1f-4f36-8ed3-12cb01bf128b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z6b8h" Oct 30 17:12:11.346610 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.346503 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2939b3a2-1950-44c9-802c-a53861aa7aa8-config-volume\") pod \"dns-default-22vlj\" (UID: \"2939b3a2-1950-44c9-802c-a53861aa7aa8\") " pod="openshift-dns/dns-default-22vlj" Oct 30 17:12:11.346610 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.346523 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csqbr\" (UniqueName: \"kubernetes.io/projected/2989c0bb-0ed6-41b2-ad3e-bb6e85bd60b9-kube-api-access-csqbr\") pod \"volume-data-source-validator-58c9f5f46d-4x66d\" (UID: \"2989c0bb-0ed6-41b2-ad3e-bb6e85bd60b9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4x66d" Oct 30 17:12:11.346610 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.346540 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-metrics-certs\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:11.346610 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.346555 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/16cd937e-4c6d-45d0-a9de-f77dc836d082-snapshots\") pod \"insights-operator-7c4986d449-4w8fl\" (UID: \"16cd937e-4c6d-45d0-a9de-f77dc836d082\") " pod="openshift-insights/insights-operator-7c4986d449-4w8fl" Oct 30 17:12:11.346610 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.346579 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-certificates\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:11.346610 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.346598 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd74ba94-86d0-4707-94e5-6822c4ab5751-trusted-ca\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:11.346610 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.346615 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx4rg\" (UniqueName: \"kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-kube-api-access-qx4rg\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:11.347207 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.346630 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16cd937e-4c6d-45d0-a9de-f77dc836d082-service-ca-bundle\") pod \"insights-operator-7c4986d449-4w8fl\" (UID: \"16cd937e-4c6d-45d0-a9de-f77dc836d082\") " pod="openshift-insights/insights-operator-7c4986d449-4w8fl" Oct 30 17:12:11.347207 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.346652 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtlv4\" (UniqueName: \"kubernetes.io/projected/16cd937e-4c6d-45d0-a9de-f77dc836d082-kube-api-access-wtlv4\") pod \"insights-operator-7c4986d449-4w8fl\" (UID: \"16cd937e-4c6d-45d0-a9de-f77dc836d082\") " pod="openshift-insights/insights-operator-7c4986d449-4w8fl" Oct 30 17:12:11.347207 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.346670 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc71466d-eac0-460b-af50-3204f07b3f35-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6jbgg\" (UID: \"cc71466d-eac0-460b-af50-3204f07b3f35\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jbgg" Oct 30 17:12:11.347207 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.346691 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd74ba94-86d0-4707-94e5-6822c4ab5751-installation-pull-secrets\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:11.347207 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.346743 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cc71466d-eac0-460b-af50-3204f07b3f35-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6jbgg\" (UID: \"cc71466d-eac0-460b-af50-3204f07b3f35\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jbgg" Oct 30 17:12:11.347207 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.346781 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16cd937e-4c6d-45d0-a9de-f77dc836d082-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4w8fl\" (UID: \"16cd937e-4c6d-45d0-a9de-f77dc836d082\") " pod="openshift-insights/insights-operator-7c4986d449-4w8fl" Oct 30 17:12:11.347207 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.346828 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-default-certificate\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:11.347207 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.346857 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/27ac623c-2f57-44ec-a1b1-3ad87909452d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jccch\" (UID: \"27ac623c-2f57-44ec-a1b1-3ad87909452d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jccch" Oct 30 17:12:11.347207 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.346961 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 17:12:11.347207 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.347011 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-metrics-certs podName:924ac066-ff98-4320-9eb2-268f0a39ca33 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:11.846998066 +0000 UTC m=+31.771500605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-metrics-certs") pod "router-default-ddb54c4f7-54tw2" (UID: "924ac066-ff98-4320-9eb2-268f0a39ca33") : secret "router-metrics-certs-default" not found Oct 30 17:12:11.347619 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.347595 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/16cd937e-4c6d-45d0-a9de-f77dc836d082-snapshots\") pod \"insights-operator-7c4986d449-4w8fl\" (UID: \"16cd937e-4c6d-45d0-a9de-f77dc836d082\") " pod="openshift-insights/insights-operator-7c4986d449-4w8fl" Oct 30 17:12:11.347707 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.347689 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16cd937e-4c6d-45d0-a9de-f77dc836d082-service-ca-bundle\") pod \"insights-operator-7c4986d449-4w8fl\" (UID: \"16cd937e-4c6d-45d0-a9de-f77dc836d082\") " pod="openshift-insights/insights-operator-7c4986d449-4w8fl" Oct 30 17:12:11.348612 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.347884 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 17:12:11.348612 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.347902 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/16cd937e-4c6d-45d0-a9de-f77dc836d082-serving-cert\") pod \"insights-operator-7c4986d449-4w8fl\" (UID: \"16cd937e-4c6d-45d0-a9de-f77dc836d082\") " pod="openshift-insights/insights-operator-7c4986d449-4w8fl" Oct 30 17:12:11.348612 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.347939 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddss9\" (UniqueName: \"kubernetes.io/projected/89eff971-c01b-4b06-9c16-4332956558a3-kube-api-access-ddss9\") pod \"ingress-canary-rdjn5\" (UID: \"89eff971-c01b-4b06-9c16-4332956558a3\") " pod="openshift-ingress-canary/ingress-canary-rdjn5" Oct 30 17:12:11.348612 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.347993 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/924ac066-ff98-4320-9eb2-268f0a39ca33-service-ca-bundle\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:11.348612 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.348019 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/195f952d-ee96-485d-99aa-8b47e6633c97-serving-cert\") pod \"service-ca-operator-544d749c8c-2tk8w\" (UID: \"195f952d-ee96-485d-99aa-8b47e6633c97\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2tk8w" Oct 30 17:12:11.348612 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.348113 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrqrz\" (UniqueName: \"kubernetes.io/projected/924ac066-ff98-4320-9eb2-268f0a39ca33-kube-api-access-mrqrz\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:11.348612 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.348164 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc71466d-eac0-460b-af50-3204f07b3f35-cluster-monitoring-operator-tls podName:cc71466d-eac0-460b-af50-3204f07b3f35 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:11.848147957 +0000 UTC m=+31.772650490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc71466d-eac0-460b-af50-3204f07b3f35-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6jbgg" (UID: "cc71466d-eac0-460b-af50-3204f07b3f35") : secret "cluster-monitoring-operator-tls" not found Oct 30 17:12:11.348612 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.348195 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l25lh\" (UniqueName: \"kubernetes.io/projected/195f952d-ee96-485d-99aa-8b47e6633c97-kube-api-access-l25lh\") pod \"service-ca-operator-544d749c8c-2tk8w\" (UID: \"195f952d-ee96-485d-99aa-8b47e6633c97\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2tk8w" Oct 30 17:12:11.348612 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.348323 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-certificates\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:11.348612 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.348482 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89eff971-c01b-4b06-9c16-4332956558a3-cert\") pod \"ingress-canary-rdjn5\" (UID: \"89eff971-c01b-4b06-9c16-4332956558a3\") " pod="openshift-ingress-canary/ingress-canary-rdjn5" Oct 30 17:12:11.348612 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.348518 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd74ba94-86d0-4707-94e5-6822c4ab5751-ca-trust-extracted\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:11.348612 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.348542 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-bound-sa-token\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:11.348612 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.348571 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd74ba94-86d0-4707-94e5-6822c4ab5751-image-registry-private-configuration\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:11.349321 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.349143 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd74ba94-86d0-4707-94e5-6822c4ab5751-trusted-ca\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:11.349321 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.349183 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16cd937e-4c6d-45d0-a9de-f77dc836d082-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4w8fl\" (UID: \"16cd937e-4c6d-45d0-a9de-f77dc836d082\") " pod="openshift-insights/insights-operator-7c4986d449-4w8fl" Oct 30 17:12:11.355940 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.349494 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd74ba94-86d0-4707-94e5-6822c4ab5751-installation-pull-secrets\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:11.355940 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.351096 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/924ac066-ff98-4320-9eb2-268f0a39ca33-service-ca-bundle podName:924ac066-ff98-4320-9eb2-268f0a39ca33 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:11.851076278 +0000 UTC m=+31.775578839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/924ac066-ff98-4320-9eb2-268f0a39ca33-service-ca-bundle") pod "router-default-ddb54c4f7-54tw2" (UID: "924ac066-ff98-4320-9eb2-268f0a39ca33") : configmap references non-existent config key: service-ca.crt Oct 30 17:12:11.355940 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.351231 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/27ac623c-2f57-44ec-a1b1-3ad87909452d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jccch\" (UID: \"27ac623c-2f57-44ec-a1b1-3ad87909452d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jccch" Oct 30 17:12:11.355940 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.351312 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-stats-auth\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:11.355940 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.351339 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/195f952d-ee96-485d-99aa-8b47e6633c97-config\") pod \"service-ca-operator-544d749c8c-2tk8w\" (UID: \"195f952d-ee96-485d-99aa-8b47e6633c97\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2tk8w" Oct 30 17:12:11.355940 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.351364 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16cd937e-4c6d-45d0-a9de-f77dc836d082-tmp\") pod \"insights-operator-7c4986d449-4w8fl\" (UID: \"16cd937e-4c6d-45d0-a9de-f77dc836d082\") " pod="openshift-insights/insights-operator-7c4986d449-4w8fl" Oct 30 17:12:11.355940 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.351399 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd74ba94-86d0-4707-94e5-6822c4ab5751-ca-trust-extracted\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:11.355940 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.351692 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16cd937e-4c6d-45d0-a9de-f77dc836d082-tmp\") pod \"insights-operator-7c4986d449-4w8fl\" (UID: \"16cd937e-4c6d-45d0-a9de-f77dc836d082\") " pod="openshift-insights/insights-operator-7c4986d449-4w8fl" Oct 30 17:12:11.355940 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.352491 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cc71466d-eac0-460b-af50-3204f07b3f35-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6jbgg\" (UID: \"cc71466d-eac0-460b-af50-3204f07b3f35\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jbgg" Oct 30 17:12:11.355940 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.352545 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/16cd937e-4c6d-45d0-a9de-f77dc836d082-serving-cert\") pod \"insights-operator-7c4986d449-4w8fl\" (UID: \"16cd937e-4c6d-45d0-a9de-f77dc836d082\") " pod="openshift-insights/insights-operator-7c4986d449-4w8fl" Oct 30 17:12:11.355940 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.352965 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/195f952d-ee96-485d-99aa-8b47e6633c97-config\") pod \"service-ca-operator-544d749c8c-2tk8w\" (UID: \"195f952d-ee96-485d-99aa-8b47e6633c97\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2tk8w" Oct 30 17:12:11.355940 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.354053 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/195f952d-ee96-485d-99aa-8b47e6633c97-serving-cert\") pod \"service-ca-operator-544d749c8c-2tk8w\" (UID: \"195f952d-ee96-485d-99aa-8b47e6633c97\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2tk8w" Oct 30 17:12:11.355940 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.354228 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-default-certificate\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:11.355940 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.355219 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd74ba94-86d0-4707-94e5-6822c4ab5751-image-registry-private-configuration\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:11.355940 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.355833 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-stats-auth\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:11.376710 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.375719 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rttdq\" (UniqueName: \"kubernetes.io/projected/cc71466d-eac0-460b-af50-3204f07b3f35-kube-api-access-rttdq\") pod \"cluster-monitoring-operator-884576c57-6jbgg\" (UID: \"cc71466d-eac0-460b-af50-3204f07b3f35\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jbgg" Oct 30 17:12:11.376710 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.375757 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtlv4\" (UniqueName: \"kubernetes.io/projected/16cd937e-4c6d-45d0-a9de-f77dc836d082-kube-api-access-wtlv4\") pod \"insights-operator-7c4986d449-4w8fl\" (UID: \"16cd937e-4c6d-45d0-a9de-f77dc836d082\") " pod="openshift-insights/insights-operator-7c4986d449-4w8fl" Oct 30 17:12:11.376710 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.375830 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx4rg\" (UniqueName: \"kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-kube-api-access-qx4rg\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:11.376710 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.376311 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csqbr\" (UniqueName: \"kubernetes.io/projected/2989c0bb-0ed6-41b2-ad3e-bb6e85bd60b9-kube-api-access-csqbr\") pod \"volume-data-source-validator-58c9f5f46d-4x66d\" (UID: \"2989c0bb-0ed6-41b2-ad3e-bb6e85bd60b9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4x66d" Oct 30 17:12:11.376710 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.376309 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrqrz\" (UniqueName: \"kubernetes.io/projected/924ac066-ff98-4320-9eb2-268f0a39ca33-kube-api-access-mrqrz\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:11.377513 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.377449 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l25lh\" (UniqueName: \"kubernetes.io/projected/195f952d-ee96-485d-99aa-8b47e6633c97-kube-api-access-l25lh\") pod \"service-ca-operator-544d749c8c-2tk8w\" (UID: \"195f952d-ee96-485d-99aa-8b47e6633c97\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2tk8w" Oct 30 17:12:11.378394 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.378362 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-bound-sa-token\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:11.451901 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.451854 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffnbr\" (UniqueName: \"kubernetes.io/projected/2939b3a2-1950-44c9-802c-a53861aa7aa8-kube-api-access-ffnbr\") pod \"dns-default-22vlj\" (UID: \"2939b3a2-1950-44c9-802c-a53861aa7aa8\") " pod="openshift-dns/dns-default-22vlj" Oct 30 17:12:11.452056 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.451920 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2939b3a2-1950-44c9-802c-a53861aa7aa8-tmp-dir\") pod \"dns-default-22vlj\" (UID: \"2939b3a2-1950-44c9-802c-a53861aa7aa8\") " pod="openshift-dns/dns-default-22vlj" Oct 30 17:12:11.452056 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.451948 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zz48\" (UniqueName: \"kubernetes.io/projected/e5f38da3-fe1f-4f36-8ed3-12cb01bf128b-kube-api-access-5zz48\") pod \"network-check-source-6f78d59f99-z6b8h\" (UID: \"e5f38da3-fe1f-4f36-8ed3-12cb01bf128b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z6b8h" Oct 30 17:12:11.452056 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.451972 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2939b3a2-1950-44c9-802c-a53861aa7aa8-config-volume\") pod \"dns-default-22vlj\" (UID: \"2939b3a2-1950-44c9-802c-a53861aa7aa8\") " pod="openshift-dns/dns-default-22vlj" Oct 30 17:12:11.452200 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.452074 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/27ac623c-2f57-44ec-a1b1-3ad87909452d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jccch\" (UID: \"27ac623c-2f57-44ec-a1b1-3ad87909452d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jccch" Oct 30 17:12:11.452200 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.452105 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddss9\" (UniqueName: \"kubernetes.io/projected/89eff971-c01b-4b06-9c16-4332956558a3-kube-api-access-ddss9\") pod \"ingress-canary-rdjn5\" (UID: \"89eff971-c01b-4b06-9c16-4332956558a3\") " pod="openshift-ingress-canary/ingress-canary-rdjn5" Oct 30 17:12:11.452200 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.452171 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89eff971-c01b-4b06-9c16-4332956558a3-cert\") pod \"ingress-canary-rdjn5\" (UID: \"89eff971-c01b-4b06-9c16-4332956558a3\") " pod="openshift-ingress-canary/ingress-canary-rdjn5" Oct 30 17:12:11.452339 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.452208 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/27ac623c-2f57-44ec-a1b1-3ad87909452d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jccch\" (UID: \"27ac623c-2f57-44ec-a1b1-3ad87909452d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jccch" Oct 30 17:12:11.452339 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.452302 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4qm2\" (UniqueName: \"kubernetes.io/projected/27ac623c-2f57-44ec-a1b1-3ad87909452d-kube-api-access-g4qm2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jccch\" (UID: \"27ac623c-2f57-44ec-a1b1-3ad87909452d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jccch" Oct 30 17:12:11.452339 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.452330 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2939b3a2-1950-44c9-802c-a53861aa7aa8-metrics-tls\") pod \"dns-default-22vlj\" (UID: \"2939b3a2-1950-44c9-802c-a53861aa7aa8\") " pod="openshift-dns/dns-default-22vlj" Oct 30 17:12:11.452478 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.452453 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:12:11.452524 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.452512 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2939b3a2-1950-44c9-802c-a53861aa7aa8-metrics-tls podName:2939b3a2-1950-44c9-802c-a53861aa7aa8 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:11.952494128 +0000 UTC m=+31.876996677 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2939b3a2-1950-44c9-802c-a53861aa7aa8-metrics-tls") pod "dns-default-22vlj" (UID: "2939b3a2-1950-44c9-802c-a53861aa7aa8") : secret "dns-default-metrics-tls" not found Oct 30 17:12:11.453308 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.452597 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2939b3a2-1950-44c9-802c-a53861aa7aa8-tmp-dir\") pod \"dns-default-22vlj\" (UID: \"2939b3a2-1950-44c9-802c-a53861aa7aa8\") " pod="openshift-dns/dns-default-22vlj" Oct 30 17:12:11.453308 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.453062 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2939b3a2-1950-44c9-802c-a53861aa7aa8-config-volume\") pod \"dns-default-22vlj\" (UID: \"2939b3a2-1950-44c9-802c-a53861aa7aa8\") " pod="openshift-dns/dns-default-22vlj" Oct 30 17:12:11.453308 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.453159 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/27ac623c-2f57-44ec-a1b1-3ad87909452d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jccch\" (UID: \"27ac623c-2f57-44ec-a1b1-3ad87909452d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jccch" Oct 30 17:12:11.456029 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.455512 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/27ac623c-2f57-44ec-a1b1-3ad87909452d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jccch\" (UID: \"27ac623c-2f57-44ec-a1b1-3ad87909452d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jccch" Oct 30 17:12:11.456562 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.456451 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89eff971-c01b-4b06-9c16-4332956558a3-cert\") pod \"ingress-canary-rdjn5\" (UID: \"89eff971-c01b-4b06-9c16-4332956558a3\") " pod="openshift-ingress-canary/ingress-canary-rdjn5" Oct 30 17:12:11.463174 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.461936 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2tk8w" Oct 30 17:12:11.479484 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.479461 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zz48\" (UniqueName: \"kubernetes.io/projected/e5f38da3-fe1f-4f36-8ed3-12cb01bf128b-kube-api-access-5zz48\") pod \"network-check-source-6f78d59f99-z6b8h\" (UID: \"e5f38da3-fe1f-4f36-8ed3-12cb01bf128b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z6b8h" Oct 30 17:12:11.479623 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.479605 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddss9\" (UniqueName: \"kubernetes.io/projected/89eff971-c01b-4b06-9c16-4332956558a3-kube-api-access-ddss9\") pod \"ingress-canary-rdjn5\" (UID: \"89eff971-c01b-4b06-9c16-4332956558a3\") " pod="openshift-ingress-canary/ingress-canary-rdjn5" Oct 30 17:12:11.479741 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.479713 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffnbr\" (UniqueName: \"kubernetes.io/projected/2939b3a2-1950-44c9-802c-a53861aa7aa8-kube-api-access-ffnbr\") pod \"dns-default-22vlj\" (UID: \"2939b3a2-1950-44c9-802c-a53861aa7aa8\") " pod="openshift-dns/dns-default-22vlj" Oct 30 17:12:11.482206 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.482181 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4qm2\" (UniqueName: \"kubernetes.io/projected/27ac623c-2f57-44ec-a1b1-3ad87909452d-kube-api-access-g4qm2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jccch\" (UID: \"27ac623c-2f57-44ec-a1b1-3ad87909452d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jccch" Oct 30 17:12:11.493959 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.493939 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-4w8fl" Oct 30 17:12:11.518275 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.518248 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4x66d" Oct 30 17:12:11.529728 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.529705 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jccch" Oct 30 17:12:11.540303 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.540251 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z6b8h" Oct 30 17:12:11.616268 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.616032 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rdjn5" Oct 30 17:12:11.624410 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.624302 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t2pqm" Oct 30 17:12:11.625009 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.624987 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:12:11.639673 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.639650 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-pnbwd\"" Oct 30 17:12:11.639781 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.639765 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-n92rc\"" Oct 30 17:12:11.640329 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.639942 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 17:12:11.654969 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.654946 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hwfbh"] Oct 30 17:12:11.658948 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.658924 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9jx9m"] Oct 30 17:12:11.659055 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.658977 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-w5rnm"] Oct 30 17:12:11.712518 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.712276 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-mf9rd"] Oct 30 17:12:11.716812 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:12:11.716779 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eab122e_07da_4307_b7f2_914cbe2b4237.slice/crio-6b5f03698ad2b8d27127136fb8ac9b2d5f1f6cfdfdc2a1cc7c2e7b8540ee4663 WatchSource:0}: Error finding container 6b5f03698ad2b8d27127136fb8ac9b2d5f1f6cfdfdc2a1cc7c2e7b8540ee4663: Status 404 returned error can't find the container with id 6b5f03698ad2b8d27127136fb8ac9b2d5f1f6cfdfdc2a1cc7c2e7b8540ee4663 Oct 30 17:12:11.755498 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.755460 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c2d3293d-1890-4fa2-b790-9f84622c9c8b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kvpw6\" (UID: \"c2d3293d-1890-4fa2-b790-9f84622c9c8b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kvpw6" Oct 30 17:12:11.755654 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.755554 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-tls\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:11.755717 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.755679 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/745f4f0f-74f7-4fff-9384-c8f45f601e33-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-t9hdn\" (UID: \"745f4f0f-74f7-4fff-9384-c8f45f601e33\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t9hdn" Oct 30 17:12:11.755840 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.755824 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 17:12:11.755944 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.755920 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/745f4f0f-74f7-4fff-9384-c8f45f601e33-samples-operator-tls podName:745f4f0f-74f7-4fff-9384-c8f45f601e33 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:12.75589951 +0000 UTC m=+32.680402060 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/745f4f0f-74f7-4fff-9384-c8f45f601e33-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-t9hdn" (UID: "745f4f0f-74f7-4fff-9384-c8f45f601e33") : secret "samples-operator-tls" not found Oct 30 17:12:11.756338 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.756317 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 17:12:11.756424 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.756377 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2d3293d-1890-4fa2-b790-9f84622c9c8b-networking-console-plugin-cert podName:c2d3293d-1890-4fa2-b790-9f84622c9c8b nodeName:}" failed. No retries permitted until 2026-10-30 17:12:12.756360436 +0000 UTC m=+32.680862971 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c2d3293d-1890-4fa2-b790-9f84622c9c8b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kvpw6" (UID: "c2d3293d-1890-4fa2-b790-9f84622c9c8b") : secret "networking-console-plugin-cert" not found Oct 30 17:12:11.756484 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.756445 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:12:11.756484 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.756456 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-585b9dc6c5-dblkr: secret "image-registry-tls" not found Oct 30 17:12:11.756590 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.756490 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-tls podName:b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce nodeName:}" failed. No retries permitted until 2026-10-30 17:12:12.756479261 +0000 UTC m=+32.680981794 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-tls") pod "image-registry-585b9dc6c5-dblkr" (UID: "b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce") : secret "image-registry-tls" not found Oct 30 17:12:11.764771 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.764273 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-2tk8w"] Oct 30 17:12:11.769012 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:12:11.768983 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195f952d_ee96_485d_99aa_8b47e6633c97.slice/crio-8f7b8781d9e26722564e26832573d120947b1c44750561ef7848fa85b315d8e9 WatchSource:0}: Error finding container 8f7b8781d9e26722564e26832573d120947b1c44750561ef7848fa85b315d8e9: Status 404 returned error can't find the container with id 8f7b8781d9e26722564e26832573d120947b1c44750561ef7848fa85b315d8e9 Oct 30 17:12:11.811454 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.811401 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-4w8fl"] Oct 30 17:12:11.815608 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:12:11.815579 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16cd937e_4c6d_45d0_a9de_f77dc836d082.slice/crio-74f894480aa0bf718f1fbba10d211a7853089a2b067b1e45fe10b6c2f05f6094 WatchSource:0}: Error finding container 74f894480aa0bf718f1fbba10d211a7853089a2b067b1e45fe10b6c2f05f6094: Status 404 returned error can't find the container with id 74f894480aa0bf718f1fbba10d211a7853089a2b067b1e45fe10b6c2f05f6094 Oct 30 17:12:11.836532 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.836506 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4x66d"] Oct 30 17:12:11.840941 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:12:11.840911 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2989c0bb_0ed6_41b2_ad3e_bb6e85bd60b9.slice/crio-9428e795c347bf480a762ae94fca6bd2f8e9a1629cb722c587615ae8ef671124 WatchSource:0}: Error finding container 9428e795c347bf480a762ae94fca6bd2f8e9a1629cb722c587615ae8ef671124: Status 404 returned error can't find the container with id 9428e795c347bf480a762ae94fca6bd2f8e9a1629cb722c587615ae8ef671124 Oct 30 17:12:11.856490 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.856462 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-tls\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:11.856611 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.856526 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-metrics-certs\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:11.856611 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.856587 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc71466d-eac0-460b-af50-3204f07b3f35-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6jbgg\" (UID: \"cc71466d-eac0-460b-af50-3204f07b3f35\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jbgg" Oct 30 17:12:11.856801 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.856613 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:12:11.856801 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.856633 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77579944f4-hctng: secret "image-registry-tls" not found Oct 30 17:12:11.856801 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.856683 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-tls podName:fd74ba94-86d0-4707-94e5-6822c4ab5751 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:12.856667627 +0000 UTC m=+32.781170160 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-tls") pod "image-registry-77579944f4-hctng" (UID: "fd74ba94-86d0-4707-94e5-6822c4ab5751") : secret "image-registry-tls" not found Oct 30 17:12:11.856801 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.856682 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 17:12:11.856801 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.856703 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/924ac066-ff98-4320-9eb2-268f0a39ca33-service-ca-bundle\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:11.856801 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.856688 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 17:12:11.856801 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.856743 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-metrics-certs podName:924ac066-ff98-4320-9eb2-268f0a39ca33 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:12.856725966 +0000 UTC m=+32.781228511 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-metrics-certs") pod "router-default-ddb54c4f7-54tw2" (UID: "924ac066-ff98-4320-9eb2-268f0a39ca33") : secret "router-metrics-certs-default" not found Oct 30 17:12:11.856801 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.856770 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc71466d-eac0-460b-af50-3204f07b3f35-cluster-monitoring-operator-tls podName:cc71466d-eac0-460b-af50-3204f07b3f35 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:12.85675765 +0000 UTC m=+32.781260218 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc71466d-eac0-460b-af50-3204f07b3f35-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6jbgg" (UID: "cc71466d-eac0-460b-af50-3204f07b3f35") : secret "cluster-monitoring-operator-tls" not found Oct 30 17:12:11.857104 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.856832 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/924ac066-ff98-4320-9eb2-268f0a39ca33-service-ca-bundle podName:924ac066-ff98-4320-9eb2-268f0a39ca33 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:12.856822302 +0000 UTC m=+32.781324871 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/924ac066-ff98-4320-9eb2-268f0a39ca33-service-ca-bundle") pod "router-default-ddb54c4f7-54tw2" (UID: "924ac066-ff98-4320-9eb2-268f0a39ca33") : configmap references non-existent config key: service-ca.crt Oct 30 17:12:11.864691 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.864659 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-z6b8h"] Oct 30 17:12:11.867936 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.867914 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jccch"] Oct 30 17:12:11.872026 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:12:11.871987 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ac623c_2f57_44ec_a1b1_3ad87909452d.slice/crio-d46a7f3e70d669027e4930adb828ba01c2c6e5c44f0a5bf10cdc0d06f1e10b38 WatchSource:0}: Error finding container d46a7f3e70d669027e4930adb828ba01c2c6e5c44f0a5bf10cdc0d06f1e10b38: Status 404 returned error can't find the container with id d46a7f3e70d669027e4930adb828ba01c2c6e5c44f0a5bf10cdc0d06f1e10b38 Oct 30 17:12:11.925649 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.925607 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rdjn5"] Oct 30 17:12:11.928748 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.928719 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z6b8h" event={"ID":"e5f38da3-fe1f-4f36-8ed3-12cb01bf128b","Type":"ContainerStarted","Data":"e8a68f579b2467a036f5512235b4620091a569e46370b57a92317184447bc764"} Oct 30 17:12:11.929075 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:12:11.929055 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89eff971_c01b_4b06_9c16_4332956558a3.slice/crio-b333be98afc85046a05ce164d47c64803e19f7598f1591771baf6ad2e71317d8 WatchSource:0}: Error finding container b333be98afc85046a05ce164d47c64803e19f7598f1591771baf6ad2e71317d8: Status 404 returned error can't find the container with id b333be98afc85046a05ce164d47c64803e19f7598f1591771baf6ad2e71317d8 Oct 30 17:12:11.930779 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.930366 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-hwfbh" event={"ID":"e30e6b59-b8e2-4125-ae7a-41edd1ce7389","Type":"ContainerStarted","Data":"d4d5e5de498e3d15b8e50f1c66beb5169c18ef10cd7187da87ad6a527c06f476"} Oct 30 17:12:11.930779 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.930417 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-hwfbh" event={"ID":"e30e6b59-b8e2-4125-ae7a-41edd1ce7389","Type":"ContainerStarted","Data":"69639bd161d7b5158a9ebe9374202d313f6c64a12bfc34b5ebe75a9030a5a215"} Oct 30 17:12:11.931877 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.931842 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4x66d" event={"ID":"2989c0bb-0ed6-41b2-ad3e-bb6e85bd60b9","Type":"ContainerStarted","Data":"9428e795c347bf480a762ae94fca6bd2f8e9a1629cb722c587615ae8ef671124"} Oct 30 17:12:11.933560 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.933535 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-9jx9m" event={"ID":"c7ac81c7-e1f0-4549-886d-d7b8b893b614","Type":"ContainerStarted","Data":"963d96a2cfa20b029ca8a7dd2173a899613e81cad200f2809f12a862c737ac5f"} Oct 30 17:12:11.933648 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.933576 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-9jx9m" event={"ID":"c7ac81c7-e1f0-4549-886d-d7b8b893b614","Type":"ContainerStarted","Data":"eefca769132f66ee6473ad8447eb6a35ff9c5c8c8b82fbccf7aaf883425b9373"} Oct 30 17:12:11.934915 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.934893 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-w5rnm" event={"ID":"03a0e7f9-9e86-4aa9-9864-205263f6a486","Type":"ContainerStarted","Data":"c5a049d774873abac1b58a614946dc8d307fa59b4fcd7d480652302aff414b52"} Oct 30 17:12:11.934987 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.934927 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-w5rnm" event={"ID":"03a0e7f9-9e86-4aa9-9864-205263f6a486","Type":"ContainerStarted","Data":"128cefb7c44adc89580c33e6ad06c84105963070eebc6e81f53281534582965b"} Oct 30 17:12:11.935996 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.935974 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jccch" event={"ID":"27ac623c-2f57-44ec-a1b1-3ad87909452d","Type":"ContainerStarted","Data":"d46a7f3e70d669027e4930adb828ba01c2c6e5c44f0a5bf10cdc0d06f1e10b38"} Oct 30 17:12:11.937054 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.937036 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4w8fl" event={"ID":"16cd937e-4c6d-45d0-a9de-f77dc836d082","Type":"ContainerStarted","Data":"74f894480aa0bf718f1fbba10d211a7853089a2b067b1e45fe10b6c2f05f6094"} Oct 30 17:12:11.938083 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.938066 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2tk8w" event={"ID":"195f952d-ee96-485d-99aa-8b47e6633c97","Type":"ContainerStarted","Data":"8f7b8781d9e26722564e26832573d120947b1c44750561ef7848fa85b315d8e9"} Oct 30 17:12:11.939042 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.939023 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" event={"ID":"1eab122e-07da-4307-b7f2-914cbe2b4237","Type":"ContainerStarted","Data":"6b5f03698ad2b8d27127136fb8ac9b2d5f1f6cfdfdc2a1cc7c2e7b8540ee4663"} Oct 30 17:12:11.958032 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:11.958011 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2939b3a2-1950-44c9-802c-a53861aa7aa8-metrics-tls\") pod \"dns-default-22vlj\" (UID: \"2939b3a2-1950-44c9-802c-a53861aa7aa8\") " pod="openshift-dns/dns-default-22vlj" Oct 30 17:12:11.958163 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.958147 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:12:11.958211 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:11.958206 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2939b3a2-1950-44c9-802c-a53861aa7aa8-metrics-tls podName:2939b3a2-1950-44c9-802c-a53861aa7aa8 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:12.95819243 +0000 UTC m=+32.882694965 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2939b3a2-1950-44c9-802c-a53861aa7aa8-metrics-tls") pod "dns-default-22vlj" (UID: "2939b3a2-1950-44c9-802c-a53861aa7aa8") : secret "dns-default-metrics-tls" not found Oct 30 17:12:12.021155 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:12.021115 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-w5rnm" podStartSLOduration=387.021103468 podStartE2EDuration="6m27.021103468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:12.02063165 +0000 UTC m=+31.945134204" watchObservedRunningTime="2026-10-30 17:12:12.021103468 +0000 UTC m=+31.945606022" Oct 30 17:12:12.021688 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:12.021665 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-hwfbh" podStartSLOduration=387.021660275 podStartE2EDuration="6m27.021660275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:11.969821561 +0000 UTC m=+31.894324112" watchObservedRunningTime="2026-10-30 17:12:12.021660275 +0000 UTC m=+31.946162828" Oct 30 17:12:12.060934 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:12.060897 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-9jx9m" podStartSLOduration=387.060885259 podStartE2EDuration="6m27.060885259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:12.060850709 +0000 UTC m=+31.985353263" watchObservedRunningTime="2026-10-30 17:12:12.060885259 +0000 UTC m=+31.985387812" Oct 30 17:12:12.623117 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:12.623067 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:12:12.639092 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:12.638097 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6mz24\"" Oct 30 17:12:12.639092 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:12.638193 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 17:12:12.766333 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:12.766295 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c2d3293d-1890-4fa2-b790-9f84622c9c8b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kvpw6\" (UID: \"c2d3293d-1890-4fa2-b790-9f84622c9c8b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kvpw6" Oct 30 17:12:12.766502 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:12.766389 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-tls\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:12.766502 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:12.766484 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/745f4f0f-74f7-4fff-9384-c8f45f601e33-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-t9hdn\" (UID: \"745f4f0f-74f7-4fff-9384-c8f45f601e33\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t9hdn" Oct 30 17:12:12.766996 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:12.766973 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 17:12:12.767098 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:12.767045 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2d3293d-1890-4fa2-b790-9f84622c9c8b-networking-console-plugin-cert podName:c2d3293d-1890-4fa2-b790-9f84622c9c8b nodeName:}" failed. No retries permitted until 2026-10-30 17:12:14.767026006 +0000 UTC m=+34.691528541 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c2d3293d-1890-4fa2-b790-9f84622c9c8b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kvpw6" (UID: "c2d3293d-1890-4fa2-b790-9f84622c9c8b") : secret "networking-console-plugin-cert" not found Oct 30 17:12:12.767476 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:12.767457 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:12:12.767533 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:12.767479 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-585b9dc6c5-dblkr: secret "image-registry-tls" not found Oct 30 17:12:12.767533 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:12.767524 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-tls podName:b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce nodeName:}" failed. No retries permitted until 2026-10-30 17:12:14.767509012 +0000 UTC m=+34.692011580 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-tls") pod "image-registry-585b9dc6c5-dblkr" (UID: "b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce") : secret "image-registry-tls" not found Oct 30 17:12:12.767626 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:12.767586 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 17:12:12.767626 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:12.767617 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/745f4f0f-74f7-4fff-9384-c8f45f601e33-samples-operator-tls podName:745f4f0f-74f7-4fff-9384-c8f45f601e33 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:14.767607604 +0000 UTC m=+34.692110141 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/745f4f0f-74f7-4fff-9384-c8f45f601e33-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-t9hdn" (UID: "745f4f0f-74f7-4fff-9384-c8f45f601e33") : secret "samples-operator-tls" not found Oct 30 17:12:12.867784 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:12.867743 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/924ac066-ff98-4320-9eb2-268f0a39ca33-service-ca-bundle\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:12.867988 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:12.867846 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-tls\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:12.867988 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:12.867941 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-metrics-certs\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:12.868101 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:12.867993 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc71466d-eac0-460b-af50-3204f07b3f35-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6jbgg\" (UID: \"cc71466d-eac0-460b-af50-3204f07b3f35\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jbgg" Oct 30 17:12:12.868160 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:12.868126 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 17:12:12.868207 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:12.868190 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc71466d-eac0-460b-af50-3204f07b3f35-cluster-monitoring-operator-tls podName:cc71466d-eac0-460b-af50-3204f07b3f35 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:14.868171719 +0000 UTC m=+34.792674257 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc71466d-eac0-460b-af50-3204f07b3f35-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6jbgg" (UID: "cc71466d-eac0-460b-af50-3204f07b3f35") : secret "cluster-monitoring-operator-tls" not found Oct 30 17:12:12.868619 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:12.868598 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/924ac066-ff98-4320-9eb2-268f0a39ca33-service-ca-bundle podName:924ac066-ff98-4320-9eb2-268f0a39ca33 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:14.868581858 +0000 UTC m=+34.793084396 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/924ac066-ff98-4320-9eb2-268f0a39ca33-service-ca-bundle") pod "router-default-ddb54c4f7-54tw2" (UID: "924ac066-ff98-4320-9eb2-268f0a39ca33") : configmap references non-existent config key: service-ca.crt Oct 30 17:12:12.868729 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:12.868685 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:12:12.868729 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:12.868696 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77579944f4-hctng: secret "image-registry-tls" not found Oct 30 17:12:12.868729 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:12.868728 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-tls podName:fd74ba94-86d0-4707-94e5-6822c4ab5751 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:14.868717999 +0000 UTC m=+34.793220532 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-tls") pod "image-registry-77579944f4-hctng" (UID: "fd74ba94-86d0-4707-94e5-6822c4ab5751") : secret "image-registry-tls" not found Oct 30 17:12:12.868926 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:12.868782 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 17:12:12.868926 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:12.868814 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-metrics-certs podName:924ac066-ff98-4320-9eb2-268f0a39ca33 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:14.868802849 +0000 UTC m=+34.793305382 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-metrics-certs") pod "router-default-ddb54c4f7-54tw2" (UID: "924ac066-ff98-4320-9eb2-268f0a39ca33") : secret "router-metrics-certs-default" not found Oct 30 17:12:12.949810 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:12.949708 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rdjn5" event={"ID":"89eff971-c01b-4b06-9c16-4332956558a3","Type":"ContainerStarted","Data":"b333be98afc85046a05ce164d47c64803e19f7598f1591771baf6ad2e71317d8"} Oct 30 17:12:12.969272 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:12.969241 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2939b3a2-1950-44c9-802c-a53861aa7aa8-metrics-tls\") pod \"dns-default-22vlj\" (UID: \"2939b3a2-1950-44c9-802c-a53861aa7aa8\") " pod="openshift-dns/dns-default-22vlj" Oct 30 17:12:12.969512 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:12.969488 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:12:12.969617 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:12.969549 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2939b3a2-1950-44c9-802c-a53861aa7aa8-metrics-tls podName:2939b3a2-1950-44c9-802c-a53861aa7aa8 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:14.96953252 +0000 UTC m=+34.894035057 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2939b3a2-1950-44c9-802c-a53861aa7aa8-metrics-tls") pod "dns-default-22vlj" (UID: "2939b3a2-1950-44c9-802c-a53861aa7aa8") : secret "dns-default-metrics-tls" not found Oct 30 17:12:13.962677 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:13.962597 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4x66d" event={"ID":"2989c0bb-0ed6-41b2-ad3e-bb6e85bd60b9","Type":"ContainerStarted","Data":"dfaa3982c09048ce702b2181b8fbbaf43bd2243726d4eb204f9d60aaf462457b"} Oct 30 17:12:14.004285 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:14.003926 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4x66d" podStartSLOduration=358.153629444 podStartE2EDuration="6m0.003906238s" totalImagesPullingTime="1.850276794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:14.002331996 +0000 UTC m=+33.926834553" watchObservedRunningTime="2026-10-30 17:12:14.003906238 +0000 UTC m=+33.928408772" Oct 30 17:12:14.385159 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:14.385115 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6178534d-9f75-49c7-a977-de65737cbcea-metrics-certs\") pod \"network-metrics-daemon-87jjv\" (UID: \"6178534d-9f75-49c7-a977-de65737cbcea\") " pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:12:14.385346 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:14.385217 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2906e95e-3baa-4c79-ad9c-b2592c1d414f-original-pull-secret\") pod \"global-pull-secret-syncer-5mtjr\" (UID: \"2906e95e-3baa-4c79-ad9c-b2592c1d414f\") " pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:12:14.385878 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:14.385837 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:12:14.385992 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:14.385933 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6178534d-9f75-49c7-a977-de65737cbcea-metrics-certs podName:6178534d-9f75-49c7-a977-de65737cbcea nodeName:}" failed. No retries permitted until 2026-10-30 17:12:46.385912758 +0000 UTC m=+66.310415305 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6178534d-9f75-49c7-a977-de65737cbcea-metrics-certs") pod "network-metrics-daemon-87jjv" (UID: "6178534d-9f75-49c7-a977-de65737cbcea") : secret "metrics-daemon-secret" not found Oct 30 17:12:14.396783 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:14.396750 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2906e95e-3baa-4c79-ad9c-b2592c1d414f-original-pull-secret\") pod \"global-pull-secret-syncer-5mtjr\" (UID: \"2906e95e-3baa-4c79-ad9c-b2592c1d414f\") " pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:12:14.486770 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:14.486720 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4njs\" (UniqueName: \"kubernetes.io/projected/d39ddd3a-781f-4084-b4e2-624b130c1025-kube-api-access-z4njs\") pod \"network-check-target-t2pqm\" (UID: \"d39ddd3a-781f-4084-b4e2-624b130c1025\") " pod="openshift-network-diagnostics/network-check-target-t2pqm" Oct 30 17:12:14.497003 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:14.496970 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4njs\" (UniqueName: \"kubernetes.io/projected/d39ddd3a-781f-4084-b4e2-624b130c1025-kube-api-access-z4njs\") pod \"network-check-target-t2pqm\" (UID: \"d39ddd3a-781f-4084-b4e2-624b130c1025\") " pod="openshift-network-diagnostics/network-check-target-t2pqm" Oct 30 17:12:14.655423 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:14.655346 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t2pqm" Oct 30 17:12:14.662574 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:14.662504 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:12:14.791190 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:14.790101 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c2d3293d-1890-4fa2-b790-9f84622c9c8b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kvpw6\" (UID: \"c2d3293d-1890-4fa2-b790-9f84622c9c8b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kvpw6" Oct 30 17:12:14.791190 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:14.790209 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-tls\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:14.791190 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:14.790324 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/745f4f0f-74f7-4fff-9384-c8f45f601e33-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-t9hdn\" (UID: \"745f4f0f-74f7-4fff-9384-c8f45f601e33\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t9hdn" Oct 30 17:12:14.791190 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:14.790473 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 17:12:14.791190 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:14.790545 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/745f4f0f-74f7-4fff-9384-c8f45f601e33-samples-operator-tls podName:745f4f0f-74f7-4fff-9384-c8f45f601e33 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:18.790524899 +0000 UTC m=+38.715027456 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/745f4f0f-74f7-4fff-9384-c8f45f601e33-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-t9hdn" (UID: "745f4f0f-74f7-4fff-9384-c8f45f601e33") : secret "samples-operator-tls" not found Oct 30 17:12:14.791190 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:14.790979 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 17:12:14.791190 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:14.791038 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2d3293d-1890-4fa2-b790-9f84622c9c8b-networking-console-plugin-cert podName:c2d3293d-1890-4fa2-b790-9f84622c9c8b nodeName:}" failed. No retries permitted until 2026-10-30 17:12:18.791021857 +0000 UTC m=+38.715524390 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c2d3293d-1890-4fa2-b790-9f84622c9c8b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kvpw6" (UID: "c2d3293d-1890-4fa2-b790-9f84622c9c8b") : secret "networking-console-plugin-cert" not found Oct 30 17:12:14.791190 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:14.791104 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:12:14.791190 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:14.791116 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-585b9dc6c5-dblkr: secret "image-registry-tls" not found Oct 30 17:12:14.791190 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:14.791151 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-tls podName:b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce nodeName:}" failed. No retries permitted until 2026-10-30 17:12:18.791140153 +0000 UTC m=+38.715642699 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-tls") pod "image-registry-585b9dc6c5-dblkr" (UID: "b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce") : secret "image-registry-tls" not found Oct 30 17:12:14.892252 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:14.891042 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-tls\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:14.892252 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:14.891152 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-metrics-certs\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:14.892252 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:14.891198 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc71466d-eac0-460b-af50-3204f07b3f35-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6jbgg\" (UID: \"cc71466d-eac0-460b-af50-3204f07b3f35\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jbgg" Oct 30 17:12:14.892252 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:14.891259 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/924ac066-ff98-4320-9eb2-268f0a39ca33-service-ca-bundle\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:14.892252 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:14.891444 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/924ac066-ff98-4320-9eb2-268f0a39ca33-service-ca-bundle podName:924ac066-ff98-4320-9eb2-268f0a39ca33 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:18.891425501 +0000 UTC m=+38.815928048 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/924ac066-ff98-4320-9eb2-268f0a39ca33-service-ca-bundle") pod "router-default-ddb54c4f7-54tw2" (UID: "924ac066-ff98-4320-9eb2-268f0a39ca33") : configmap references non-existent config key: service-ca.crt Oct 30 17:12:14.892252 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:14.891858 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:12:14.892252 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:14.891894 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77579944f4-hctng: secret "image-registry-tls" not found Oct 30 17:12:14.892252 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:14.891941 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-tls podName:fd74ba94-86d0-4707-94e5-6822c4ab5751 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:18.891925653 +0000 UTC m=+38.816428201 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-tls") pod "image-registry-77579944f4-hctng" (UID: "fd74ba94-86d0-4707-94e5-6822c4ab5751") : secret "image-registry-tls" not found Oct 30 17:12:14.892252 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:14.892010 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 17:12:14.892252 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:14.892042 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-metrics-certs podName:924ac066-ff98-4320-9eb2-268f0a39ca33 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:18.892031925 +0000 UTC m=+38.816534472 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-metrics-certs") pod "router-default-ddb54c4f7-54tw2" (UID: "924ac066-ff98-4320-9eb2-268f0a39ca33") : secret "router-metrics-certs-default" not found Oct 30 17:12:14.892252 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:14.892097 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 17:12:14.892252 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:14.892129 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc71466d-eac0-460b-af50-3204f07b3f35-cluster-monitoring-operator-tls podName:cc71466d-eac0-460b-af50-3204f07b3f35 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:18.892118559 +0000 UTC m=+38.816621105 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc71466d-eac0-460b-af50-3204f07b3f35-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6jbgg" (UID: "cc71466d-eac0-460b-af50-3204f07b3f35") : secret "cluster-monitoring-operator-tls" not found Oct 30 17:12:14.992734 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:14.992057 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2939b3a2-1950-44c9-802c-a53861aa7aa8-metrics-tls\") pod \"dns-default-22vlj\" (UID: \"2939b3a2-1950-44c9-802c-a53861aa7aa8\") " pod="openshift-dns/dns-default-22vlj" Oct 30 17:12:14.992734 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:14.992285 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:12:14.992734 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:14.992351 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2939b3a2-1950-44c9-802c-a53861aa7aa8-metrics-tls podName:2939b3a2-1950-44c9-802c-a53861aa7aa8 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:18.992330056 +0000 UTC m=+38.916832587 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2939b3a2-1950-44c9-802c-a53861aa7aa8-metrics-tls") pod "dns-default-22vlj" (UID: "2939b3a2-1950-44c9-802c-a53861aa7aa8") : secret "dns-default-metrics-tls" not found Oct 30 17:12:15.005079 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:15.004961 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t2pqm"] Oct 30 17:12:15.010214 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:12:15.010172 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd39ddd3a_781f_4084_b4e2_624b130c1025.slice/crio-43de827525a68f6af4bfe173cd354ea23de185a40e5eda53bba8309420b7571c WatchSource:0}: Error finding container 43de827525a68f6af4bfe173cd354ea23de185a40e5eda53bba8309420b7571c: Status 404 returned error can't find the container with id 43de827525a68f6af4bfe173cd354ea23de185a40e5eda53bba8309420b7571c Oct 30 17:12:15.036319 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:15.036288 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5mtjr"] Oct 30 17:12:15.043162 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:12:15.043134 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2906e95e_3baa_4c79_ad9c_b2592c1d414f.slice/crio-caa5716795758214e8a79342a54eedf4ec7ced420359018bbd53847e75eb49a3 WatchSource:0}: Error finding container caa5716795758214e8a79342a54eedf4ec7ced420359018bbd53847e75eb49a3: Status 404 returned error can't find the container with id caa5716795758214e8a79342a54eedf4ec7ced420359018bbd53847e75eb49a3 Oct 30 17:12:15.974535 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:15.974450 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5mtjr" event={"ID":"2906e95e-3baa-4c79-ad9c-b2592c1d414f","Type":"ContainerStarted","Data":"caa5716795758214e8a79342a54eedf4ec7ced420359018bbd53847e75eb49a3"} Oct 30 17:12:15.976524 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:15.976455 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t2pqm" event={"ID":"d39ddd3a-781f-4084-b4e2-624b130c1025","Type":"ContainerStarted","Data":"43de827525a68f6af4bfe173cd354ea23de185a40e5eda53bba8309420b7571c"} Oct 30 17:12:16.982811 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:16.982733 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4w8fl" event={"ID":"16cd937e-4c6d-45d0-a9de-f77dc836d082","Type":"ContainerStarted","Data":"d0ac680e4381e32ffe18de9c6cd9a7238a418be517a8c6fa8c6dd0d288673733"} Oct 30 17:12:16.986799 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:16.986771 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2tk8w" event={"ID":"195f952d-ee96-485d-99aa-8b47e6633c97","Type":"ContainerStarted","Data":"56d1a55ac073af357f593423a43c0b46cc7800e73d98aab7166b82638dc2d4cd"} Oct 30 17:12:16.990038 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:16.990016 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/0.log" Oct 30 17:12:16.990153 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:16.990057 2572 generic.go:356] "Generic (PLEG): container finished" podID="1eab122e-07da-4307-b7f2-914cbe2b4237" containerID="cd4292f808f3af5df4b30ba3eb21e3f8df9ce2bd88f164744f8c96e9ead5999a" exitCode=255 Oct 30 17:12:16.990153 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:16.990089 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" event={"ID":"1eab122e-07da-4307-b7f2-914cbe2b4237","Type":"ContainerDied","Data":"cd4292f808f3af5df4b30ba3eb21e3f8df9ce2bd88f164744f8c96e9ead5999a"} Oct 30 17:12:16.990396 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:16.990380 2572 scope.go:122] "RemoveContainer" containerID="cd4292f808f3af5df4b30ba3eb21e3f8df9ce2bd88f164744f8c96e9ead5999a" Oct 30 17:12:17.036694 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.034652 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-4w8fl" podStartSLOduration=358.554841709 podStartE2EDuration="6m3.034634228s" totalImagesPullingTime="4.479792519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:17.033473978 +0000 UTC m=+36.957976549" watchObservedRunningTime="2026-10-30 17:12:17.034634228 +0000 UTC m=+36.959136783" Oct 30 17:12:17.311755 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.311694 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2tk8w" podStartSLOduration=358.376462161 podStartE2EDuration="6m3.311666718s" totalImagesPullingTime="4.935204557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:17.145577163 +0000 UTC m=+37.070079718" watchObservedRunningTime="2026-10-30 17:12:17.311666718 +0000 UTC m=+37.236169273" Oct 30 17:12:17.313784 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.313754 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tbqhc"] Oct 30 17:12:17.336761 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.336625 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:17.337511 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.337312 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tbqhc"] Oct 30 17:12:17.352726 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.352695 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 17:12:17.353479 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.353396 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 17:12:17.353711 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.353675 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wklnr\"" Oct 30 17:12:17.397992 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.397830 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-2ftd7-xc6xx"] Oct 30 17:12:17.416994 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.416964 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-2ftd7-xc6xx" Oct 30 17:12:17.422035 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.422005 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-2ftd7-xc6xx"] Oct 30 17:12:17.519638 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.519603 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13444fae-8e71-41ee-8489-55b8606492fb-service-ca-bundle\") pod \"periodic-gathering-2ftd7-xc6xx\" (UID: \"13444fae-8e71-41ee-8489-55b8606492fb\") " pod="openshift-insights/periodic-gathering-2ftd7-xc6xx" Oct 30 17:12:17.519800 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.519645 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6084e9b4-6b79-46ab-91cf-8d292d1f4591-data-volume\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:17.519854 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.519828 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6084e9b4-6b79-46ab-91cf-8d292d1f4591-crio-socket\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:17.519937 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.519898 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/13444fae-8e71-41ee-8489-55b8606492fb-archives-path\") pod \"periodic-gathering-2ftd7-xc6xx\" (UID: \"13444fae-8e71-41ee-8489-55b8606492fb\") " pod="openshift-insights/periodic-gathering-2ftd7-xc6xx" Oct 30 17:12:17.519987 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.519943 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13444fae-8e71-41ee-8489-55b8606492fb-trusted-ca-bundle\") pod \"periodic-gathering-2ftd7-xc6xx\" (UID: \"13444fae-8e71-41ee-8489-55b8606492fb\") " pod="openshift-insights/periodic-gathering-2ftd7-xc6xx" Oct 30 17:12:17.520030 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.520014 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drkzm\" (UniqueName: \"kubernetes.io/projected/13444fae-8e71-41ee-8489-55b8606492fb-kube-api-access-drkzm\") pod \"periodic-gathering-2ftd7-xc6xx\" (UID: \"13444fae-8e71-41ee-8489-55b8606492fb\") " pod="openshift-insights/periodic-gathering-2ftd7-xc6xx" Oct 30 17:12:17.520067 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.520050 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zxfh\" (UniqueName: \"kubernetes.io/projected/6084e9b4-6b79-46ab-91cf-8d292d1f4591-kube-api-access-9zxfh\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:17.520137 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.520109 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6084e9b4-6b79-46ab-91cf-8d292d1f4591-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:17.520166 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.520143 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6084e9b4-6b79-46ab-91cf-8d292d1f4591-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:17.621321 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.621284 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/13444fae-8e71-41ee-8489-55b8606492fb-archives-path\") pod \"periodic-gathering-2ftd7-xc6xx\" (UID: \"13444fae-8e71-41ee-8489-55b8606492fb\") " pod="openshift-insights/periodic-gathering-2ftd7-xc6xx" Oct 30 17:12:17.621471 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.621337 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13444fae-8e71-41ee-8489-55b8606492fb-trusted-ca-bundle\") pod \"periodic-gathering-2ftd7-xc6xx\" (UID: \"13444fae-8e71-41ee-8489-55b8606492fb\") " pod="openshift-insights/periodic-gathering-2ftd7-xc6xx" Oct 30 17:12:17.621471 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.621396 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drkzm\" (UniqueName: \"kubernetes.io/projected/13444fae-8e71-41ee-8489-55b8606492fb-kube-api-access-drkzm\") pod \"periodic-gathering-2ftd7-xc6xx\" (UID: \"13444fae-8e71-41ee-8489-55b8606492fb\") " pod="openshift-insights/periodic-gathering-2ftd7-xc6xx" Oct 30 17:12:17.621471 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.621431 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zxfh\" (UniqueName: \"kubernetes.io/projected/6084e9b4-6b79-46ab-91cf-8d292d1f4591-kube-api-access-9zxfh\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:17.621619 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.621484 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6084e9b4-6b79-46ab-91cf-8d292d1f4591-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:17.621619 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.621504 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6084e9b4-6b79-46ab-91cf-8d292d1f4591-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:17.621619 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.621524 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13444fae-8e71-41ee-8489-55b8606492fb-service-ca-bundle\") pod \"periodic-gathering-2ftd7-xc6xx\" (UID: \"13444fae-8e71-41ee-8489-55b8606492fb\") " pod="openshift-insights/periodic-gathering-2ftd7-xc6xx" Oct 30 17:12:17.621619 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.621541 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6084e9b4-6b79-46ab-91cf-8d292d1f4591-data-volume\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:17.621760 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.621638 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6084e9b4-6b79-46ab-91cf-8d292d1f4591-crio-socket\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:17.621760 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.621731 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6084e9b4-6b79-46ab-91cf-8d292d1f4591-crio-socket\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:17.623379 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.622105 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/13444fae-8e71-41ee-8489-55b8606492fb-archives-path\") pod \"periodic-gathering-2ftd7-xc6xx\" (UID: \"13444fae-8e71-41ee-8489-55b8606492fb\") " pod="openshift-insights/periodic-gathering-2ftd7-xc6xx" Oct 30 17:12:17.623379 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.622945 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13444fae-8e71-41ee-8489-55b8606492fb-trusted-ca-bundle\") pod \"periodic-gathering-2ftd7-xc6xx\" (UID: \"13444fae-8e71-41ee-8489-55b8606492fb\") " pod="openshift-insights/periodic-gathering-2ftd7-xc6xx" Oct 30 17:12:17.623379 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:17.623161 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:12:17.623379 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:17.623233 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6084e9b4-6b79-46ab-91cf-8d292d1f4591-insights-runtime-extractor-tls podName:6084e9b4-6b79-46ab-91cf-8d292d1f4591 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:18.12321387 +0000 UTC m=+38.047716406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6084e9b4-6b79-46ab-91cf-8d292d1f4591-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbqhc" (UID: "6084e9b4-6b79-46ab-91cf-8d292d1f4591") : secret "insights-runtime-extractor-tls" not found Oct 30 17:12:17.623379 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.623309 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6084e9b4-6b79-46ab-91cf-8d292d1f4591-data-volume\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:17.623670 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:17.623649 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:12:17.623741 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:17.623709 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6084e9b4-6b79-46ab-91cf-8d292d1f4591-kube-rbac-proxy-cm podName:6084e9b4-6b79-46ab-91cf-8d292d1f4591 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:18.123693115 +0000 UTC m=+38.048195661 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6084e9b4-6b79-46ab-91cf-8d292d1f4591-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tbqhc" (UID: "6084e9b4-6b79-46ab-91cf-8d292d1f4591") : configmap "kube-rbac-proxy" not found Oct 30 17:12:17.623741 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.623650 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13444fae-8e71-41ee-8489-55b8606492fb-service-ca-bundle\") pod \"periodic-gathering-2ftd7-xc6xx\" (UID: \"13444fae-8e71-41ee-8489-55b8606492fb\") " pod="openshift-insights/periodic-gathering-2ftd7-xc6xx" Oct 30 17:12:17.657627 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.657598 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drkzm\" (UniqueName: \"kubernetes.io/projected/13444fae-8e71-41ee-8489-55b8606492fb-kube-api-access-drkzm\") pod \"periodic-gathering-2ftd7-xc6xx\" (UID: \"13444fae-8e71-41ee-8489-55b8606492fb\") " pod="openshift-insights/periodic-gathering-2ftd7-xc6xx" Oct 30 17:12:17.658949 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.658875 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zxfh\" (UniqueName: \"kubernetes.io/projected/6084e9b4-6b79-46ab-91cf-8d292d1f4591-kube-api-access-9zxfh\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:17.747700 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.747653 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-2ftd7-xc6xx" Oct 30 17:12:17.998199 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:17.997423 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jccch" event={"ID":"27ac623c-2f57-44ec-a1b1-3ad87909452d","Type":"ContainerStarted","Data":"c59e356e22f1cf78c847667f8dba99875135754baf2de47ada4ba1413447079d"} Oct 30 17:12:18.000379 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.000352 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 17:12:18.001741 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.000815 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/0.log" Oct 30 17:12:18.001741 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.000852 2572 generic.go:356] "Generic (PLEG): container finished" podID="1eab122e-07da-4307-b7f2-914cbe2b4237" containerID="c562428fe21e43cf22e26ebeba52550c57e2741dc839171b41e59c2d589d4922" exitCode=255 Oct 30 17:12:18.001741 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.000936 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" event={"ID":"1eab122e-07da-4307-b7f2-914cbe2b4237","Type":"ContainerDied","Data":"c562428fe21e43cf22e26ebeba52550c57e2741dc839171b41e59c2d589d4922"} Oct 30 17:12:18.001741 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.000991 2572 scope.go:122] "RemoveContainer" containerID="cd4292f808f3af5df4b30ba3eb21e3f8df9ce2bd88f164744f8c96e9ead5999a" Oct 30 17:12:18.001741 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.001337 2572 scope.go:122] "RemoveContainer" containerID="c562428fe21e43cf22e26ebeba52550c57e2741dc839171b41e59c2d589d4922" Oct 30 17:12:18.001741 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:18.001515 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-mf9rd_openshift-console-operator(1eab122e-07da-4307-b7f2-914cbe2b4237)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" podUID="1eab122e-07da-4307-b7f2-914cbe2b4237" Oct 30 17:12:18.004580 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.004555 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rdjn5" event={"ID":"89eff971-c01b-4b06-9c16-4332956558a3","Type":"ContainerStarted","Data":"a603ac5d4572491c8a64003a7a5dd34aa1f309eca1407bdeb4e21b31d66d05a8"} Oct 30 17:12:18.042288 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.042233 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jccch" podStartSLOduration=358.862295263 podStartE2EDuration="6m4.042215451s" totalImagesPullingTime="5.179920188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:18.041035508 +0000 UTC m=+37.965538061" watchObservedRunningTime="2026-10-30 17:12:18.042215451 +0000 UTC m=+37.966718004" Oct 30 17:12:18.097147 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.096690 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-2ftd7-xc6xx"] Oct 30 17:12:18.127628 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.127572 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rdjn5" podStartSLOduration=2.516782667 podStartE2EDuration="8.127556828s" totalImagesPullingTime="5.610774161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:18.126284587 +0000 UTC m=+38.050787155" watchObservedRunningTime="2026-10-30 17:12:18.127556828 +0000 UTC m=+38.052059424" Oct 30 17:12:18.127902 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.127873 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6084e9b4-6b79-46ab-91cf-8d292d1f4591-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:18.128010 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.127954 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6084e9b4-6b79-46ab-91cf-8d292d1f4591-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:18.129420 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:18.129392 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:12:18.129520 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:18.129467 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6084e9b4-6b79-46ab-91cf-8d292d1f4591-insights-runtime-extractor-tls podName:6084e9b4-6b79-46ab-91cf-8d292d1f4591 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:19.129451138 +0000 UTC m=+39.053953697 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6084e9b4-6b79-46ab-91cf-8d292d1f4591-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbqhc" (UID: "6084e9b4-6b79-46ab-91cf-8d292d1f4591") : secret "insights-runtime-extractor-tls" not found Oct 30 17:12:18.129896 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:18.129878 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:12:18.129962 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:18.129949 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6084e9b4-6b79-46ab-91cf-8d292d1f4591-kube-rbac-proxy-cm podName:6084e9b4-6b79-46ab-91cf-8d292d1f4591 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:19.129931302 +0000 UTC m=+39.054433863 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6084e9b4-6b79-46ab-91cf-8d292d1f4591-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tbqhc" (UID: "6084e9b4-6b79-46ab-91cf-8d292d1f4591") : configmap "kube-rbac-proxy" not found Oct 30 17:12:18.234667 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.234635 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-27cwx"] Oct 30 17:12:18.257321 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.257291 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-27cwx"] Oct 30 17:12:18.257449 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.257384 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-27cwx" Oct 30 17:12:18.272328 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.272308 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 17:12:18.272475 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.272387 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 17:12:18.272475 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.272393 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 17:12:18.272591 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.272565 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 17:12:18.272591 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.272581 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jj2qr\"" Oct 30 17:12:18.275757 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:12:18.275730 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13444fae_8e71_41ee_8489_55b8606492fb.slice/crio-837040638b1b42bf20445cb551dc04c6277f9fbef3a55987538fc7f0b9bf8c15 WatchSource:0}: Error finding container 837040638b1b42bf20445cb551dc04c6277f9fbef3a55987538fc7f0b9bf8c15: Status 404 returned error can't find the container with id 837040638b1b42bf20445cb551dc04c6277f9fbef3a55987538fc7f0b9bf8c15 Oct 30 17:12:18.430465 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.430148 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4bdc401b-ae3f-48c0-8110-8cd6096007d8-signing-cabundle\") pod \"service-ca-5fffc97bdf-27cwx\" (UID: \"4bdc401b-ae3f-48c0-8110-8cd6096007d8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-27cwx" Oct 30 17:12:18.430465 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.430229 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4bdc401b-ae3f-48c0-8110-8cd6096007d8-signing-key\") pod \"service-ca-5fffc97bdf-27cwx\" (UID: \"4bdc401b-ae3f-48c0-8110-8cd6096007d8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-27cwx" Oct 30 17:12:18.430465 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.430342 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkvxq\" (UniqueName: \"kubernetes.io/projected/4bdc401b-ae3f-48c0-8110-8cd6096007d8-kube-api-access-xkvxq\") pod \"service-ca-5fffc97bdf-27cwx\" (UID: \"4bdc401b-ae3f-48c0-8110-8cd6096007d8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-27cwx" Oct 30 17:12:18.531614 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.531577 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4bdc401b-ae3f-48c0-8110-8cd6096007d8-signing-key\") pod \"service-ca-5fffc97bdf-27cwx\" (UID: \"4bdc401b-ae3f-48c0-8110-8cd6096007d8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-27cwx" Oct 30 17:12:18.531773 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.531735 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkvxq\" (UniqueName: \"kubernetes.io/projected/4bdc401b-ae3f-48c0-8110-8cd6096007d8-kube-api-access-xkvxq\") pod \"service-ca-5fffc97bdf-27cwx\" (UID: \"4bdc401b-ae3f-48c0-8110-8cd6096007d8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-27cwx" Oct 30 17:12:18.531968 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.531920 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4bdc401b-ae3f-48c0-8110-8cd6096007d8-signing-cabundle\") pod \"service-ca-5fffc97bdf-27cwx\" (UID: \"4bdc401b-ae3f-48c0-8110-8cd6096007d8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-27cwx" Oct 30 17:12:18.532733 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.532688 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4bdc401b-ae3f-48c0-8110-8cd6096007d8-signing-cabundle\") pod \"service-ca-5fffc97bdf-27cwx\" (UID: \"4bdc401b-ae3f-48c0-8110-8cd6096007d8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-27cwx" Oct 30 17:12:18.534485 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.534460 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4bdc401b-ae3f-48c0-8110-8cd6096007d8-signing-key\") pod \"service-ca-5fffc97bdf-27cwx\" (UID: \"4bdc401b-ae3f-48c0-8110-8cd6096007d8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-27cwx" Oct 30 17:12:18.555509 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.555484 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkvxq\" (UniqueName: \"kubernetes.io/projected/4bdc401b-ae3f-48c0-8110-8cd6096007d8-kube-api-access-xkvxq\") pod \"service-ca-5fffc97bdf-27cwx\" (UID: \"4bdc401b-ae3f-48c0-8110-8cd6096007d8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-27cwx" Oct 30 17:12:18.580428 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.580401 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-27cwx" Oct 30 17:12:18.836768 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.835777 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/745f4f0f-74f7-4fff-9384-c8f45f601e33-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-t9hdn\" (UID: \"745f4f0f-74f7-4fff-9384-c8f45f601e33\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t9hdn" Oct 30 17:12:18.836768 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.835897 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c2d3293d-1890-4fa2-b790-9f84622c9c8b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kvpw6\" (UID: \"c2d3293d-1890-4fa2-b790-9f84622c9c8b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kvpw6" Oct 30 17:12:18.836768 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.835956 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-tls\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:18.836768 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:18.836112 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:12:18.836768 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:18.836124 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-585b9dc6c5-dblkr: secret "image-registry-tls" not found Oct 30 17:12:18.836768 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:18.836172 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-tls podName:b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce nodeName:}" failed. No retries permitted until 2026-10-30 17:12:26.836159774 +0000 UTC m=+46.760662320 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-tls") pod "image-registry-585b9dc6c5-dblkr" (UID: "b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce") : secret "image-registry-tls" not found Oct 30 17:12:18.836768 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:18.836558 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 17:12:18.836768 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:18.836617 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/745f4f0f-74f7-4fff-9384-c8f45f601e33-samples-operator-tls podName:745f4f0f-74f7-4fff-9384-c8f45f601e33 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:26.836602031 +0000 UTC m=+46.761104584 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/745f4f0f-74f7-4fff-9384-c8f45f601e33-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-t9hdn" (UID: "745f4f0f-74f7-4fff-9384-c8f45f601e33") : secret "samples-operator-tls" not found Oct 30 17:12:18.836768 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:18.836683 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 17:12:18.836768 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:18.836730 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2d3293d-1890-4fa2-b790-9f84622c9c8b-networking-console-plugin-cert podName:c2d3293d-1890-4fa2-b790-9f84622c9c8b nodeName:}" failed. No retries permitted until 2026-10-30 17:12:26.836708722 +0000 UTC m=+46.761211254 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c2d3293d-1890-4fa2-b790-9f84622c9c8b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kvpw6" (UID: "c2d3293d-1890-4fa2-b790-9f84622c9c8b") : secret "networking-console-plugin-cert" not found Oct 30 17:12:18.909351 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.909273 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-27cwx"] Oct 30 17:12:18.914002 ip-10-0-139-222 kubenswrapper[2572]: W1030 17:12:18.913974 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bdc401b_ae3f_48c0_8110_8cd6096007d8.slice/crio-4d7da57858bd0907a716dbaac42f1783b0ace83acd5f89ab37219848d1b5d0d9 WatchSource:0}: Error finding container 4d7da57858bd0907a716dbaac42f1783b0ace83acd5f89ab37219848d1b5d0d9: Status 404 returned error can't find the container with id 4d7da57858bd0907a716dbaac42f1783b0ace83acd5f89ab37219848d1b5d0d9 Oct 30 17:12:18.936331 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.936307 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/924ac066-ff98-4320-9eb2-268f0a39ca33-service-ca-bundle\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:18.936441 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.936417 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-tls\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:18.936513 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.936501 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-metrics-certs\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:18.936564 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:18.936547 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc71466d-eac0-460b-af50-3204f07b3f35-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6jbgg\" (UID: \"cc71466d-eac0-460b-af50-3204f07b3f35\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jbgg" Oct 30 17:12:18.936733 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:18.936713 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/924ac066-ff98-4320-9eb2-268f0a39ca33-service-ca-bundle podName:924ac066-ff98-4320-9eb2-268f0a39ca33 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:26.936697689 +0000 UTC m=+46.861200220 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/924ac066-ff98-4320-9eb2-268f0a39ca33-service-ca-bundle") pod "router-default-ddb54c4f7-54tw2" (UID: "924ac066-ff98-4320-9eb2-268f0a39ca33") : configmap references non-existent config key: service-ca.crt Oct 30 17:12:18.936733 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:18.936734 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 17:12:18.936905 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:18.936790 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc71466d-eac0-460b-af50-3204f07b3f35-cluster-monitoring-operator-tls podName:cc71466d-eac0-460b-af50-3204f07b3f35 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:26.936773932 +0000 UTC m=+46.861276477 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc71466d-eac0-460b-af50-3204f07b3f35-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6jbgg" (UID: "cc71466d-eac0-460b-af50-3204f07b3f35") : secret "cluster-monitoring-operator-tls" not found Oct 30 17:12:18.936905 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:18.936812 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:12:18.936905 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:18.936831 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77579944f4-hctng: secret "image-registry-tls" not found Oct 30 17:12:18.936905 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:18.936900 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-tls podName:fd74ba94-86d0-4707-94e5-6822c4ab5751 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:26.936884052 +0000 UTC m=+46.861386590 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-tls") pod "image-registry-77579944f4-hctng" (UID: "fd74ba94-86d0-4707-94e5-6822c4ab5751") : secret "image-registry-tls" not found Oct 30 17:12:18.937091 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:18.936909 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 17:12:18.937091 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:18.936936 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-metrics-certs podName:924ac066-ff98-4320-9eb2-268f0a39ca33 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:26.936927123 +0000 UTC m=+46.861429687 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-metrics-certs") pod "router-default-ddb54c4f7-54tw2" (UID: "924ac066-ff98-4320-9eb2-268f0a39ca33") : secret "router-metrics-certs-default" not found Oct 30 17:12:19.010696 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:19.010671 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 17:12:19.011204 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:19.011017 2572 scope.go:122] "RemoveContainer" containerID="c562428fe21e43cf22e26ebeba52550c57e2741dc839171b41e59c2d589d4922" Oct 30 17:12:19.011264 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:19.011221 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-mf9rd_openshift-console-operator(1eab122e-07da-4307-b7f2-914cbe2b4237)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" podUID="1eab122e-07da-4307-b7f2-914cbe2b4237" Oct 30 17:12:19.012829 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:19.012804 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z6b8h" event={"ID":"e5f38da3-fe1f-4f36-8ed3-12cb01bf128b","Type":"ContainerStarted","Data":"ca3c996cecb2562f03f756d9215e35f17fe2ce48a736e70d5a14f89659761cf7"} Oct 30 17:12:19.015898 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:19.015375 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t2pqm" event={"ID":"d39ddd3a-781f-4084-b4e2-624b130c1025","Type":"ContainerStarted","Data":"5a67dbb071cac2c13ee3fd77db3e399d3d52620834e7f513ebe5c1a30c64eecd"} Oct 30 17:12:19.015898 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:19.015834 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t2pqm" Oct 30 17:12:19.018359 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:19.018333 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-27cwx" event={"ID":"4bdc401b-ae3f-48c0-8110-8cd6096007d8","Type":"ContainerStarted","Data":"193f7bdbb71faaf9369d4514feefb023808f5731a5d5b12c16ccbaf8f4a35ffe"} Oct 30 17:12:19.018438 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:19.018373 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-27cwx" event={"ID":"4bdc401b-ae3f-48c0-8110-8cd6096007d8","Type":"ContainerStarted","Data":"4d7da57858bd0907a716dbaac42f1783b0ace83acd5f89ab37219848d1b5d0d9"} Oct 30 17:12:19.019932 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:19.019912 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-2ftd7-xc6xx" event={"ID":"13444fae-8e71-41ee-8489-55b8606492fb","Type":"ContainerStarted","Data":"7b0c8c13a33067c6eb1c2d86c8e6da418f167cf984bd042570c78809302d4d34"} Oct 30 17:12:19.020005 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:19.019948 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-2ftd7-xc6xx" event={"ID":"13444fae-8e71-41ee-8489-55b8606492fb","Type":"ContainerStarted","Data":"837040638b1b42bf20445cb551dc04c6277f9fbef3a55987538fc7f0b9bf8c15"} Oct 30 17:12:19.038197 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:19.038162 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2939b3a2-1950-44c9-802c-a53861aa7aa8-metrics-tls\") pod \"dns-default-22vlj\" (UID: \"2939b3a2-1950-44c9-802c-a53861aa7aa8\") " pod="openshift-dns/dns-default-22vlj" Oct 30 17:12:19.038335 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:19.038317 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:12:19.038415 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:19.038389 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2939b3a2-1950-44c9-802c-a53861aa7aa8-metrics-tls podName:2939b3a2-1950-44c9-802c-a53861aa7aa8 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:27.038369807 +0000 UTC m=+46.962872339 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2939b3a2-1950-44c9-802c-a53861aa7aa8-metrics-tls") pod "dns-default-22vlj" (UID: "2939b3a2-1950-44c9-802c-a53861aa7aa8") : secret "dns-default-metrics-tls" not found Oct 30 17:12:19.096680 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:19.096629 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-27cwx" podStartSLOduration=1.096616285 podStartE2EDuration="1.096616285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:19.096404857 +0000 UTC m=+39.020907411" watchObservedRunningTime="2026-10-30 17:12:19.096616285 +0000 UTC m=+39.021118840" Oct 30 17:12:19.138983 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:19.138914 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6084e9b4-6b79-46ab-91cf-8d292d1f4591-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:19.140157 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:19.139002 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6084e9b4-6b79-46ab-91cf-8d292d1f4591-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:19.140157 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:19.139576 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:12:19.140157 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:19.139641 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6084e9b4-6b79-46ab-91cf-8d292d1f4591-kube-rbac-proxy-cm podName:6084e9b4-6b79-46ab-91cf-8d292d1f4591 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:21.139623034 +0000 UTC m=+41.064125569 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6084e9b4-6b79-46ab-91cf-8d292d1f4591-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tbqhc" (UID: "6084e9b4-6b79-46ab-91cf-8d292d1f4591") : configmap "kube-rbac-proxy" not found Oct 30 17:12:19.140157 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:19.139946 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:12:19.140157 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:19.140011 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6084e9b4-6b79-46ab-91cf-8d292d1f4591-insights-runtime-extractor-tls podName:6084e9b4-6b79-46ab-91cf-8d292d1f4591 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:21.139995155 +0000 UTC m=+41.064497687 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6084e9b4-6b79-46ab-91cf-8d292d1f4591-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbqhc" (UID: "6084e9b4-6b79-46ab-91cf-8d292d1f4591") : secret "insights-runtime-extractor-tls" not found Oct 30 17:12:19.142892 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:19.142372 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z6b8h" podStartSLOduration=352.685949941 podStartE2EDuration="5m59.142354856s" totalImagesPullingTime="6.456404915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:19.141974896 +0000 UTC m=+39.066477451" watchObservedRunningTime="2026-10-30 17:12:19.142354856 +0000 UTC m=+39.066857410" Oct 30 17:12:19.184485 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:19.184389 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-t2pqm" podStartSLOduration=34.593902638 podStartE2EDuration="38.184371036s" totalImagesPullingTime="3.590468398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:19.183248735 +0000 UTC m=+39.107751291" watchObservedRunningTime="2026-10-30 17:12:19.184371036 +0000 UTC m=+39.108873591" Oct 30 17:12:19.226405 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:19.226352 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-2ftd7-xc6xx" podStartSLOduration=2.226334331 podStartE2EDuration="2.226334331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:19.226043677 +0000 UTC m=+39.150546231" watchObservedRunningTime="2026-10-30 17:12:19.226334331 +0000 UTC m=+39.150836886" Oct 30 17:12:20.098872 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:20.098823 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zgb2b_467187ee-7039-4bb4-adbe-38161ee7c336/dns-node-resolver/0.log" Oct 30 17:12:21.062910 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:21.062881 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-stqrx_2664a881-44d7-470a-93d0-df1b3eaca0bc/node-ca/0.log" Oct 30 17:12:21.161791 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:21.161760 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6084e9b4-6b79-46ab-91cf-8d292d1f4591-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:21.162202 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:21.161797 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6084e9b4-6b79-46ab-91cf-8d292d1f4591-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:21.162202 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:21.161888 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:12:21.162202 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:21.161918 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:12:21.162202 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:21.161943 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6084e9b4-6b79-46ab-91cf-8d292d1f4591-kube-rbac-proxy-cm podName:6084e9b4-6b79-46ab-91cf-8d292d1f4591 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:25.161929456 +0000 UTC m=+45.086432005 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6084e9b4-6b79-46ab-91cf-8d292d1f4591-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tbqhc" (UID: "6084e9b4-6b79-46ab-91cf-8d292d1f4591") : configmap "kube-rbac-proxy" not found Oct 30 17:12:21.162202 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:21.161981 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6084e9b4-6b79-46ab-91cf-8d292d1f4591-insights-runtime-extractor-tls podName:6084e9b4-6b79-46ab-91cf-8d292d1f4591 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:25.161967624 +0000 UTC m=+45.086470183 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6084e9b4-6b79-46ab-91cf-8d292d1f4591-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbqhc" (UID: "6084e9b4-6b79-46ab-91cf-8d292d1f4591") : secret "insights-runtime-extractor-tls" not found Oct 30 17:12:21.342204 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:21.342166 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" Oct 30 17:12:21.342342 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:21.342246 2572 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" Oct 30 17:12:21.342584 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:21.342572 2572 scope.go:122] "RemoveContainer" containerID="c562428fe21e43cf22e26ebeba52550c57e2741dc839171b41e59c2d589d4922" Oct 30 17:12:21.342769 ip-10-0-139-222 kubenswrapper[2572]: E1030 17:12:21.342750 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-mf9rd_openshift-console-operator(1eab122e-07da-4307-b7f2-914cbe2b4237)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" podUID="1eab122e-07da-4307-b7f2-914cbe2b4237" Oct 30 17:12:21.657478 ip-10-0-139-222 kubenswrapper[2572]: I1030 17:12:21.657451 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rdjn5_89eff971-c01b-4b06-9c16-4332956558a3/serve-healthcheck-canary/0.log" Oct 30 17:12:21.879447 ip-10-0-139-222 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 17:12:21.894274 ip-10-0-139-222 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 17:12:21.894484 ip-10-0-139-222 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 17:12:21.894543 ip-10-0-139-222 systemd[1]: kubelet.service: Consumed 3.020s CPU time. Oct 30 17:12:21.908459 ip-10-0-139-222 systemd[1]: Starting Kubernetes Kubelet... Oct 30 17:12:22.011653 ip-10-0-139-222 kubenswrapper[4965]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:12:22.011653 ip-10-0-139-222 kubenswrapper[4965]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 17:12:22.011653 ip-10-0-139-222 kubenswrapper[4965]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:12:22.011653 ip-10-0-139-222 kubenswrapper[4965]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:12:22.013684 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013601 4965 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:12:22.013684 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013685 4965 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:12:22.013757 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013689 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:12:22.013757 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013693 4965 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:12:22.013757 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013696 4965 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:12:22.013757 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013698 4965 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:12:22.013757 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013701 4965 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:12:22.013757 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013704 4965 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:12:22.013757 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013706 4965 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:12:22.013757 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013709 4965 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:12:22.013757 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013711 4965 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:12:22.013757 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013714 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:12:22.013757 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013717 4965 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:12:22.013757 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013719 4965 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:12:22.013757 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013722 4965 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:12:22.013757 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013730 4965 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:12:22.013757 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013733 4965 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:12:22.013757 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013738 4965 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:12:22.013757 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013742 4965 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:12:22.013757 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013745 4965 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:12:22.013757 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013749 4965 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:12:22.014267 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013752 4965 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:12:22.014267 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013756 4965 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:12:22.014267 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013759 4965 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:12:22.014267 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013762 4965 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:12:22.014267 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013765 4965 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:12:22.014267 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013768 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:12:22.014267 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013771 4965 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:12:22.014267 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013774 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:12:22.014267 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013776 4965 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:12:22.014267 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013779 4965 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:12:22.014267 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013782 4965 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:12:22.014267 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013784 4965 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:12:22.014267 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013787 4965 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:12:22.014267 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013789 4965 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:12:22.014267 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013792 4965 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:12:22.014267 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013794 4965 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:12:22.014267 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013797 4965 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:12:22.014267 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013799 4965 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:12:22.014267 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013802 4965 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:12:22.014267 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013804 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:12:22.014778 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013807 4965 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:12:22.014778 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013809 4965 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:12:22.014778 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013811 4965 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:12:22.014778 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013814 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:12:22.014778 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013817 4965 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:12:22.014778 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013820 4965 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:12:22.014778 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013823 4965 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:12:22.014778 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013826 4965 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:12:22.014778 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013828 4965 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:12:22.014778 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013831 4965 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:12:22.014778 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013833 4965 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:12:22.014778 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013836 4965 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:12:22.014778 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013839 4965 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:12:22.014778 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013842 4965 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:12:22.014778 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013844 4965 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:12:22.014778 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013847 4965 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:12:22.014778 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013849 4965 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:12:22.014778 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013852 4965 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:12:22.014778 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013855 4965 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:12:22.015294 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013857 4965 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:12:22.015294 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013872 4965 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:12:22.015294 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013876 4965 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:12:22.015294 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013879 4965 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:12:22.015294 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013882 4965 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:12:22.015294 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013884 4965 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:12:22.015294 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013887 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:12:22.015294 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013889 4965 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:12:22.015294 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013892 4965 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:12:22.015294 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013895 4965 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:12:22.015294 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013897 4965 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:12:22.015294 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013900 4965 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:12:22.015294 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013903 4965 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:12:22.015294 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013906 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:12:22.015294 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013910 4965 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:12:22.015294 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013913 4965 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:12:22.015294 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013916 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:12:22.015294 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013919 4965 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:12:22.015294 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013922 4965 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:12:22.015772 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013924 4965 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:12:22.015772 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013927 4965 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:12:22.015772 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013930 4965 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:12:22.015772 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013933 4965 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:12:22.015772 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013935 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:12:22.015772 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013938 4965 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:12:22.015772 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013940 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:12:22.015772 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013943 4965 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:12:22.015772 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013946 4965 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:12:22.015772 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013949 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:12:22.015772 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013952 4965 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:12:22.015772 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013954 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:12:22.015772 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013957 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:12:22.015772 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013960 4965 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:12:22.015772 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013962 4965 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:12:22.015772 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013965 4965 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:12:22.015772 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013967 4965 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:12:22.015772 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013970 4965 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:12:22.015772 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013973 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:12:22.015772 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013976 4965 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:12:22.016322 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013979 4965 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:12:22.016322 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013981 4965 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:12:22.016322 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013984 4965 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:12:22.016322 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013986 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:12:22.016322 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013989 4965 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:12:22.016322 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013992 4965 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:12:22.016322 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.013995 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:12:22.016322 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014754 4965 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:12:22.016322 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014761 4965 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:12:22.016322 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014764 4965 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:12:22.016322 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014766 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:12:22.016322 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014769 4965 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:12:22.016322 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014772 4965 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:12:22.016322 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014775 4965 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:12:22.016322 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014778 4965 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:12:22.016322 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014781 4965 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:12:22.016322 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014784 4965 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:12:22.016322 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014787 4965 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:12:22.016322 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014790 4965 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:12:22.016322 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014793 4965 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:12:22.016833 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014795 4965 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:12:22.016833 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014798 4965 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:12:22.016833 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014802 4965 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:12:22.016833 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014806 4965 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:12:22.016833 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014809 4965 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:12:22.016833 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014811 4965 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:12:22.016833 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014814 4965 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:12:22.016833 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014817 4965 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:12:22.016833 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014819 4965 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:12:22.016833 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014822 4965 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:12:22.016833 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014825 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:12:22.016833 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014828 4965 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:12:22.016833 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014830 4965 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:12:22.016833 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014833 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:12:22.016833 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014837 4965 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:12:22.016833 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014839 4965 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:12:22.016833 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014842 4965 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:12:22.016833 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014845 4965 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:12:22.016833 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014847 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:12:22.017323 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014850 4965 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:12:22.017323 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014852 4965 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:12:22.017323 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014855 4965 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:12:22.017323 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014857 4965 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:12:22.017323 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014873 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:12:22.017323 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014876 4965 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:12:22.017323 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014878 4965 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:12:22.017323 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014881 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:12:22.017323 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014884 4965 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:12:22.017323 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014887 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:12:22.017323 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014890 4965 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:12:22.017323 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014892 4965 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:12:22.017323 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014895 4965 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:12:22.017323 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014897 4965 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:12:22.017323 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014900 4965 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:12:22.017323 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014903 4965 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:12:22.017323 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014906 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:12:22.017323 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014909 4965 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:12:22.017323 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014911 4965 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:12:22.017323 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014914 4965 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:12:22.017812 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014918 4965 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:12:22.017812 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014921 4965 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:12:22.017812 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014923 4965 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:12:22.017812 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014926 4965 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:12:22.017812 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014928 4965 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:12:22.017812 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014931 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:12:22.017812 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014934 4965 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:12:22.017812 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014937 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:12:22.017812 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014940 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:12:22.017812 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014942 4965 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:12:22.017812 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014945 4965 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:12:22.017812 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014947 4965 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:12:22.017812 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014950 4965 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:12:22.017812 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014952 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:12:22.017812 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014955 4965 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:12:22.017812 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014957 4965 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:12:22.017812 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014960 4965 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:12:22.017812 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014962 4965 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:12:22.017812 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014965 4965 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:12:22.018310 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014968 4965 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:12:22.018310 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014970 4965 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:12:22.018310 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014975 4965 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:12:22.018310 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014979 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:12:22.018310 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014983 4965 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:12:22.018310 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014986 4965 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:12:22.018310 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014990 4965 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:12:22.018310 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014993 4965 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:12:22.018310 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.014997 4965 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:12:22.018310 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015002 4965 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:12:22.018310 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015005 4965 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:12:22.018310 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015008 4965 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:12:22.018310 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015012 4965 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:12:22.018310 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015017 4965 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:12:22.018310 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015020 4965 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:12:22.018310 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015023 4965 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:12:22.018310 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015025 4965 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:12:22.018310 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015028 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:12:22.018310 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015033 4965 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:12:22.018783 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015036 4965 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:12:22.018783 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015039 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:12:22.018783 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015041 4965 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:12:22.018783 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015044 4965 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:12:22.018783 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015047 4965 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:12:22.018783 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015050 4965 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:12:22.018783 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015053 4965 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:12:22.018783 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015055 4965 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:12:22.018783 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015057 4965 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:12:22.018783 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015060 4965 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:12:22.018783 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015063 4965 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:12:22.018783 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015065 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:12:22.018783 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015068 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:12:22.018783 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015070 4965 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:12:22.018783 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015074 4965 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:12:22.018783 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015090 4965 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:12:22.018783 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015630 4965 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:12:22.018783 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015636 4965 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:12:22.018783 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015639 4965 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:12:22.019307 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015642 4965 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:12:22.019307 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015645 4965 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:12:22.019307 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015650 4965 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:12:22.019307 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015653 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:12:22.019307 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015656 4965 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:12:22.019307 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015659 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:12:22.019307 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015661 4965 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:12:22.019307 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015664 4965 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:12:22.019307 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015667 4965 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:12:22.019307 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015669 4965 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:12:22.019307 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015672 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:12:22.019307 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015674 4965 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:12:22.019307 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015679 4965 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:12:22.019307 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015682 4965 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:12:22.019307 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015684 4965 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:12:22.019307 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015687 4965 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:12:22.019307 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015689 4965 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:12:22.019307 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015692 4965 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:12:22.019307 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015694 4965 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:12:22.019307 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015697 4965 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:12:22.020166 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015699 4965 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:12:22.020166 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015702 4965 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:12:22.020166 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015705 4965 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:12:22.020166 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015707 4965 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:12:22.020166 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015710 4965 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:12:22.020166 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015713 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:12:22.020166 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015715 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:12:22.020166 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015718 4965 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:12:22.020166 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015720 4965 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:12:22.020166 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015724 4965 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:12:22.020166 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015727 4965 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:12:22.020166 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015729 4965 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:12:22.020166 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015732 4965 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:12:22.020166 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015734 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:12:22.020166 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015738 4965 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:12:22.020166 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015741 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:12:22.020166 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015745 4965 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:12:22.020166 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015748 4965 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:12:22.020166 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015751 4965 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:12:22.021021 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015755 4965 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:12:22.021021 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015758 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:12:22.021021 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015760 4965 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:12:22.021021 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015763 4965 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:12:22.021021 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015765 4965 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:12:22.021021 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015769 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:12:22.021021 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015772 4965 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:12:22.021021 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015774 4965 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:12:22.021021 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015777 4965 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:12:22.021021 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015780 4965 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:12:22.021021 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015783 4965 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:12:22.021021 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015786 4965 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:12:22.021021 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015789 4965 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:12:22.021021 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015791 4965 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:12:22.021021 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015794 4965 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:12:22.021021 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015796 4965 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:12:22.021021 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015799 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:12:22.021021 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015801 4965 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:12:22.021021 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015804 4965 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:12:22.021021 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015806 4965 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:12:22.021964 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015809 4965 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:12:22.021964 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015811 4965 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:12:22.021964 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015814 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:12:22.021964 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015817 4965 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:12:22.021964 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015820 4965 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:12:22.021964 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015823 4965 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:12:22.021964 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015827 4965 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:12:22.021964 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015831 4965 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:12:22.021964 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015834 4965 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:12:22.021964 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015837 4965 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:12:22.021964 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015839 4965 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:12:22.021964 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015842 4965 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:12:22.021964 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015844 4965 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:12:22.021964 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015847 4965 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:12:22.021964 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015849 4965 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:12:22.021964 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015852 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:12:22.021964 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015855 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:12:22.021964 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015875 4965 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:12:22.021964 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015878 4965 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:12:22.022816 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015880 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:12:22.022816 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015883 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:12:22.022816 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015886 4965 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:12:22.022816 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015888 4965 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:12:22.022816 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015891 4965 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:12:22.022816 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015894 4965 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:12:22.022816 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015896 4965 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:12:22.022816 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015899 4965 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:12:22.022816 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015902 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:12:22.022816 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015904 4965 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:12:22.022816 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015907 4965 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:12:22.022816 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015909 4965 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:12:22.022816 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015912 4965 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:12:22.022816 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015914 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:12:22.022816 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015917 4965 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:12:22.022816 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015919 4965 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:12:22.022816 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015922 4965 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:12:22.022816 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015925 4965 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:12:22.022816 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015928 4965 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:12:22.023665 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015931 4965 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:12:22.023665 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015935 4965 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:12:22.023665 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015938 4965 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:12:22.023665 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015941 4965 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:12:22.023665 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015944 4965 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:12:22.023665 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.015946 4965 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:12:22.023665 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016037 4965 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 17:12:22.023665 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016044 4965 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 17:12:22.023665 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016050 4965 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 17:12:22.023665 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016055 4965 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 17:12:22.023665 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016060 4965 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 17:12:22.023665 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016065 4965 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 17:12:22.023665 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016072 4965 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 17:12:22.023665 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016076 4965 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 17:12:22.023665 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016079 4965 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 17:12:22.023665 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016083 4965 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 17:12:22.023665 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016086 4965 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 17:12:22.023665 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016090 4965 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 17:12:22.023665 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016093 4965 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 17:12:22.023665 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016096 4965 flags.go:64] FLAG: --cgroup-root="" Oct 30 17:12:22.023665 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016099 4965 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 17:12:22.023665 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016102 4965 flags.go:64] FLAG: --client-ca-file="" Oct 30 17:12:22.023665 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016105 4965 flags.go:64] FLAG: --cloud-provider="external" Oct 30 17:12:22.024593 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016108 4965 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 17:12:22.024593 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016112 4965 flags.go:64] FLAG: --cluster-domain="" Oct 30 17:12:22.024593 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016115 4965 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 17:12:22.024593 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016118 4965 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 17:12:22.024593 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016121 4965 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 17:12:22.024593 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016125 4965 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 17:12:22.024593 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016129 4965 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 17:12:22.024593 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016132 4965 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 17:12:22.024593 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016136 4965 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 17:12:22.024593 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016139 4965 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 17:12:22.024593 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016143 4965 flags.go:64] FLAG: --contention-profiling="false" Oct 30 17:12:22.024593 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016147 4965 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 17:12:22.024593 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016150 4965 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 17:12:22.024593 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016154 4965 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 17:12:22.024593 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016157 4965 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 17:12:22.024593 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016161 4965 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 17:12:22.024593 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016164 4965 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 17:12:22.024593 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016167 4965 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 17:12:22.024593 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016170 4965 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 17:12:22.024593 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016173 4965 flags.go:64] FLAG: --enable-server="true" Oct 30 17:12:22.024593 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016179 4965 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 17:12:22.024593 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016182 4965 flags.go:64] FLAG: --event-burst="100" Oct 30 17:12:22.024593 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016186 4965 flags.go:64] FLAG: --event-qps="50" Oct 30 17:12:22.024593 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016189 4965 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 17:12:22.025490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016192 4965 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 17:12:22.025490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016195 4965 flags.go:64] FLAG: --eviction-hard="" Oct 30 17:12:22.025490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016199 4965 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 17:12:22.025490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016202 4965 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 17:12:22.025490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016205 4965 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 17:12:22.025490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016208 4965 flags.go:64] FLAG: --eviction-soft="" Oct 30 17:12:22.025490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016212 4965 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 17:12:22.025490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016215 4965 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 17:12:22.025490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016218 4965 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 17:12:22.025490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016221 4965 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 17:12:22.025490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016224 4965 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 17:12:22.025490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016227 4965 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 17:12:22.025490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016230 4965 flags.go:64] FLAG: --feature-gates="" Oct 30 17:12:22.025490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016234 4965 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 17:12:22.025490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016237 4965 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 17:12:22.025490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016240 4965 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 17:12:22.025490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016243 4965 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 17:12:22.025490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016247 4965 flags.go:64] FLAG: --healthz-port="10248" Oct 30 17:12:22.025490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016252 4965 flags.go:64] FLAG: --help="false" Oct 30 17:12:22.025490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016255 4965 flags.go:64] FLAG: --hostname-override="ip-10-0-139-222.ec2.internal" Oct 30 17:12:22.025490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016258 4965 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 17:12:22.025490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016261 4965 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 17:12:22.025490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016264 4965 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 17:12:22.026172 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016268 4965 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 17:12:22.026172 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016271 4965 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 17:12:22.026172 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016274 4965 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 17:12:22.026172 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016277 4965 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 17:12:22.026172 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016280 4965 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 17:12:22.026172 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016284 4965 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 17:12:22.026172 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016287 4965 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 17:12:22.026172 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016291 4965 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 17:12:22.026172 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016293 4965 flags.go:64] FLAG: --kube-reserved="" Oct 30 17:12:22.026172 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016297 4965 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 17:12:22.026172 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016299 4965 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 17:12:22.026172 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016303 4965 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 17:12:22.026172 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016305 4965 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 17:12:22.026172 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016309 4965 flags.go:64] FLAG: --lock-file="" Oct 30 17:12:22.026172 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016311 4965 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 17:12:22.026172 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016314 4965 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 17:12:22.026172 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016317 4965 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 17:12:22.026172 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016323 4965 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 17:12:22.026172 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016326 4965 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 17:12:22.026172 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016330 4965 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 17:12:22.026172 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016333 4965 flags.go:64] FLAG: --logging-format="text" Oct 30 17:12:22.026172 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016336 4965 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 17:12:22.026172 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016340 4965 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 17:12:22.026172 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016342 4965 flags.go:64] FLAG: --manifest-url="" Oct 30 17:12:22.026883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016345 4965 flags.go:64] FLAG: --manifest-url-header="" Oct 30 17:12:22.026883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016350 4965 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 17:12:22.026883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016353 4965 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 17:12:22.026883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016358 4965 flags.go:64] FLAG: --max-pods="110" Oct 30 17:12:22.026883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016361 4965 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 17:12:22.026883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016364 4965 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 17:12:22.026883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016367 4965 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 17:12:22.026883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016370 4965 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 17:12:22.026883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016373 4965 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 17:12:22.026883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016376 4965 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 17:12:22.026883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016379 4965 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 17:12:22.026883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016386 4965 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 17:12:22.026883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016389 4965 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 17:12:22.026883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016397 4965 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 17:12:22.026883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016400 4965 flags.go:64] FLAG: --pod-cidr="" Oct 30 17:12:22.026883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016404 4965 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 17:12:22.026883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016406 4965 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 17:12:22.026883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016409 4965 flags.go:64] FLAG: --pods-per-core="0" Oct 30 17:12:22.026883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016412 4965 flags.go:64] FLAG: --port="10250" Oct 30 17:12:22.026883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016415 4965 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 17:12:22.026883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016419 4965 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d22446e589206b6a" Oct 30 17:12:22.026883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016422 4965 flags.go:64] FLAG: --qos-reserved="" Oct 30 17:12:22.026883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016425 4965 flags.go:64] FLAG: --read-only-port="10255" Oct 30 17:12:22.026883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016428 4965 flags.go:64] FLAG: --register-node="true" Oct 30 17:12:22.026883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016431 4965 flags.go:64] FLAG: --register-with-taints="" Oct 30 17:12:22.026883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016435 4965 flags.go:64] FLAG: --registry-burst="10" Oct 30 17:12:22.027630 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016438 4965 flags.go:64] FLAG: --registry-qps="5" Oct 30 17:12:22.027630 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016441 4965 flags.go:64] FLAG: --reserved-cpus="" Oct 30 17:12:22.027630 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016444 4965 flags.go:64] FLAG: --reserved-memory="" Oct 30 17:12:22.027630 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016448 4965 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 17:12:22.027630 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016451 4965 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 17:12:22.027630 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016454 4965 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 17:12:22.027630 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016457 4965 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 17:12:22.027630 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016460 4965 flags.go:64] FLAG: --runonce="false" Oct 30 17:12:22.027630 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016463 4965 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 17:12:22.027630 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016466 4965 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 17:12:22.027630 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016470 4965 flags.go:64] FLAG: --seccomp-default="false" Oct 30 17:12:22.027630 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016473 4965 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 17:12:22.027630 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016476 4965 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 17:12:22.027630 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016479 4965 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 17:12:22.027630 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016482 4965 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 17:12:22.027630 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016485 4965 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 17:12:22.027630 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016488 4965 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 17:12:22.027630 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016491 4965 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 17:12:22.027630 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016494 4965 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 17:12:22.027630 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016498 4965 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 17:12:22.027630 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016501 4965 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 17:12:22.027630 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016504 4965 flags.go:64] FLAG: --system-cgroups="" Oct 30 17:12:22.027630 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016507 4965 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 17:12:22.027630 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016512 4965 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 17:12:22.027630 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016515 4965 flags.go:64] FLAG: --tls-cert-file="" Oct 30 17:12:22.028625 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016517 4965 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 17:12:22.028625 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016521 4965 flags.go:64] FLAG: --tls-min-version="" Oct 30 17:12:22.028625 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016524 4965 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 17:12:22.028625 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016527 4965 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 17:12:22.028625 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016530 4965 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 17:12:22.028625 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016533 4965 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 17:12:22.028625 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016536 4965 flags.go:64] FLAG: --v="2" Oct 30 17:12:22.028625 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016540 4965 flags.go:64] FLAG: --version="false" Oct 30 17:12:22.028625 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016544 4965 flags.go:64] FLAG: --vmodule="" Oct 30 17:12:22.028625 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016548 4965 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 17:12:22.028625 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.016551 4965 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 17:12:22.028625 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016687 4965 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:12:22.028625 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016691 4965 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:12:22.028625 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016694 4965 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:12:22.028625 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016697 4965 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:12:22.028625 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016700 4965 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:12:22.028625 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016702 4965 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:12:22.028625 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016706 4965 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:12:22.028625 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016710 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:12:22.028625 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016713 4965 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:12:22.028625 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016716 4965 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:12:22.028625 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016719 4965 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:12:22.029402 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016721 4965 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:12:22.029402 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016724 4965 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:12:22.029402 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016726 4965 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:12:22.029402 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016728 4965 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:12:22.029402 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016734 4965 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:12:22.029402 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016737 4965 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:12:22.029402 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016739 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:12:22.029402 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016742 4965 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:12:22.029402 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016744 4965 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:12:22.029402 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016747 4965 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:12:22.029402 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016749 4965 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:12:22.029402 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016752 4965 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:12:22.029402 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016754 4965 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:12:22.029402 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016757 4965 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:12:22.029402 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016760 4965 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:12:22.029402 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016762 4965 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:12:22.029402 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016765 4965 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:12:22.029402 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016767 4965 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:12:22.029402 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016770 4965 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:12:22.029402 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016772 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:12:22.030153 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016775 4965 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:12:22.030153 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016777 4965 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:12:22.030153 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016780 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:12:22.030153 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016782 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:12:22.030153 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016785 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:12:22.030153 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016787 4965 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:12:22.030153 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016790 4965 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:12:22.030153 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016792 4965 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:12:22.030153 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016795 4965 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:12:22.030153 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016798 4965 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:12:22.030153 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016800 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:12:22.030153 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016803 4965 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:12:22.030153 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016805 4965 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:12:22.030153 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016808 4965 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:12:22.030153 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016810 4965 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:12:22.030153 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016813 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:12:22.030153 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016816 4965 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:12:22.030153 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016819 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:12:22.030153 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016822 4965 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:12:22.030742 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016824 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:12:22.030742 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016827 4965 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:12:22.030742 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016829 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:12:22.030742 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016833 4965 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:12:22.030742 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016836 4965 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:12:22.030742 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016839 4965 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:12:22.030742 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016841 4965 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:12:22.030742 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016844 4965 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:12:22.030742 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016847 4965 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:12:22.030742 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016850 4965 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:12:22.030742 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016853 4965 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:12:22.030742 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016855 4965 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:12:22.030742 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016870 4965 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:12:22.030742 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016875 4965 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:12:22.030742 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016879 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:12:22.030742 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016882 4965 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:12:22.030742 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016884 4965 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:12:22.030742 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016888 4965 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:12:22.030742 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016890 4965 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:12:22.030742 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016893 4965 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:12:22.031618 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016895 4965 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:12:22.031618 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016898 4965 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:12:22.031618 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016900 4965 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:12:22.031618 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016904 4965 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:12:22.031618 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016906 4965 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:12:22.031618 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016909 4965 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:12:22.031618 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016911 4965 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:12:22.031618 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016914 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:12:22.031618 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016917 4965 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:12:22.031618 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016920 4965 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:12:22.031618 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016923 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:12:22.031618 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016925 4965 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:12:22.031618 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016928 4965 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:12:22.031618 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016930 4965 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:12:22.031618 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016933 4965 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:12:22.031618 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016937 4965 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:12:22.031618 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016940 4965 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:12:22.031618 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016943 4965 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:12:22.031618 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016945 4965 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:12:22.032223 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016948 4965 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:12:22.032223 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016951 4965 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:12:22.032223 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016953 4965 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:12:22.032223 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016956 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:12:22.032223 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016958 4965 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:12:22.032223 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016961 4965 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:12:22.032223 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016963 4965 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:12:22.032223 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016966 4965 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:12:22.032223 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016969 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:12:22.032223 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016971 4965 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:12:22.032223 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016974 4965 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:12:22.032223 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016977 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:12:22.032223 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016979 4965 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:12:22.032223 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016981 4965 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:12:22.032223 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016984 4965 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:12:22.032223 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016987 4965 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:12:22.032223 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.016990 4965 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:12:22.032637 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.017016 4965 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:12:22.032637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017052 4965 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:12:22.032637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017056 4965 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:12:22.032637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017059 4965 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:12:22.032637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017062 4965 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:12:22.032637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017065 4965 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:12:22.032637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017068 4965 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:12:22.032637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017072 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:12:22.032637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017075 4965 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:12:22.032637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017077 4965 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:12:22.032637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017080 4965 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:12:22.032637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017082 4965 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:12:22.032637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017086 4965 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:12:22.032637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017089 4965 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:12:22.032637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017091 4965 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:12:22.032637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017094 4965 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:12:22.033088 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017096 4965 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:12:22.033088 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017099 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:12:22.033088 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017101 4965 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:12:22.033088 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017104 4965 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:12:22.033088 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017107 4965 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:12:22.033088 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017109 4965 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:12:22.033088 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017112 4965 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:12:22.033088 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017114 4965 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:12:22.033088 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017117 4965 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:12:22.033088 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017120 4965 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:12:22.033088 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017122 4965 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:12:22.033088 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017125 4965 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:12:22.033088 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017127 4965 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:12:22.033088 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017130 4965 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:12:22.033088 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017133 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:12:22.033088 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017135 4965 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:12:22.033088 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017138 4965 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:12:22.033088 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017141 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:12:22.033088 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017143 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:12:22.033088 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017146 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:12:22.033592 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017148 4965 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:12:22.033592 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017151 4965 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:12:22.033592 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017153 4965 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:12:22.033592 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017156 4965 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:12:22.033592 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017158 4965 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:12:22.033592 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017161 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:12:22.033592 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017164 4965 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:12:22.033592 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017166 4965 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:12:22.033592 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017170 4965 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:12:22.033592 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017173 4965 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:12:22.033592 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017176 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:12:22.033592 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017178 4965 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:12:22.033592 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017181 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:12:22.033592 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017183 4965 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:12:22.033592 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017186 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:12:22.033592 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017188 4965 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:12:22.033592 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017191 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:12:22.033592 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017194 4965 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:12:22.033592 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017196 4965 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:12:22.034149 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017199 4965 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:12:22.034149 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017201 4965 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:12:22.034149 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017204 4965 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:12:22.034149 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017206 4965 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:12:22.034149 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017209 4965 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:12:22.034149 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017212 4965 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:12:22.034149 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017214 4965 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:12:22.034149 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017217 4965 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:12:22.034149 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017221 4965 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:12:22.034149 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017224 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:12:22.034149 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017227 4965 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:12:22.034149 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017234 4965 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:12:22.034149 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017237 4965 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:12:22.034149 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017239 4965 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:12:22.034149 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017242 4965 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:12:22.034149 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017244 4965 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:12:22.034149 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017247 4965 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:12:22.034149 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017249 4965 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:12:22.034149 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017252 4965 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:12:22.034709 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017254 4965 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:12:22.034709 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017257 4965 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:12:22.034709 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017261 4965 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:12:22.034709 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017264 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:12:22.034709 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017266 4965 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:12:22.034709 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017269 4965 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:12:22.034709 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017271 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:12:22.034709 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017274 4965 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:12:22.034709 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017277 4965 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:12:22.034709 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017281 4965 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:12:22.034709 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017284 4965 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:12:22.034709 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017287 4965 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:12:22.034709 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017290 4965 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:12:22.034709 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017292 4965 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:12:22.034709 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017295 4965 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:12:22.034709 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017297 4965 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:12:22.034709 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017300 4965 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:12:22.034709 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017302 4965 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:12:22.034709 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017305 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:12:22.034709 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017307 4965 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:12:22.035217 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017310 4965 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:12:22.035217 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017312 4965 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:12:22.035217 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017315 4965 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:12:22.035217 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017318 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:12:22.035217 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017320 4965 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:12:22.035217 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017324 4965 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:12:22.035217 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017327 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:12:22.035217 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017329 4965 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:12:22.035217 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017332 4965 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:12:22.035217 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017335 4965 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:12:22.035217 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017339 4965 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:12:22.035217 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017342 4965 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:12:22.035217 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.017344 4965 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:12:22.035217 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.022199 4965 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 17:12:22.035217 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.022218 4965 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 17:12:22.035217 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.022244 4965 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 17:12:22.035217 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.022251 4965 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 17:12:22.035217 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022335 4965 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:12:22.035217 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022345 4965 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:12:22.035672 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022350 4965 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:12:22.035672 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022355 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:12:22.035672 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022360 4965 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:12:22.035672 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022365 4965 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:12:22.035672 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022369 4965 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:12:22.035672 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022374 4965 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:12:22.035672 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022378 4965 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:12:22.035672 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022383 4965 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:12:22.035672 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022388 4965 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:12:22.035672 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022393 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:12:22.035672 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022398 4965 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:12:22.035672 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022402 4965 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:12:22.035672 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022406 4965 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:12:22.035672 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022410 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:12:22.035672 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022414 4965 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:12:22.035672 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022419 4965 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:12:22.035672 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022423 4965 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:12:22.035672 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022428 4965 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:12:22.035672 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022433 4965 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:12:22.035672 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022437 4965 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:12:22.036184 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022442 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:12:22.036184 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022447 4965 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:12:22.036184 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022451 4965 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:12:22.036184 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022455 4965 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:12:22.036184 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022459 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:12:22.036184 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022464 4965 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:12:22.036184 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022468 4965 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:12:22.036184 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022473 4965 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:12:22.036184 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022477 4965 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:12:22.036184 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022482 4965 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:12:22.036184 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022486 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:12:22.036184 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022491 4965 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:12:22.036184 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022495 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:12:22.036184 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022500 4965 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:12:22.036184 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022504 4965 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:12:22.036184 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022508 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:12:22.036184 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022513 4965 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:12:22.036184 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022517 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:12:22.036184 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022521 4965 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:12:22.036637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022526 4965 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:12:22.036637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022530 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:12:22.036637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022535 4965 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:12:22.036637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022539 4965 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:12:22.036637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022544 4965 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:12:22.036637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022548 4965 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:12:22.036637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022552 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:12:22.036637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022556 4965 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:12:22.036637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022561 4965 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:12:22.036637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022566 4965 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:12:22.036637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022570 4965 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:12:22.036637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022574 4965 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:12:22.036637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022579 4965 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:12:22.036637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022584 4965 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:12:22.036637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022588 4965 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:12:22.036637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022592 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:12:22.036637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022597 4965 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:12:22.036637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022601 4965 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:12:22.036637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022606 4965 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:12:22.036637 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022611 4965 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:12:22.037147 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022615 4965 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:12:22.037147 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022620 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:12:22.037147 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022624 4965 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:12:22.037147 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022628 4965 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:12:22.037147 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022633 4965 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:12:22.037147 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022637 4965 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:12:22.037147 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022641 4965 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:12:22.037147 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022646 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:12:22.037147 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022650 4965 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:12:22.037147 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022654 4965 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:12:22.037147 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022658 4965 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:12:22.037147 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022663 4965 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:12:22.037147 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022667 4965 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:12:22.037147 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022671 4965 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:12:22.037147 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022676 4965 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:12:22.037147 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022680 4965 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:12:22.037147 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022684 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:12:22.037147 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022691 4965 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:12:22.037147 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022697 4965 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:12:22.037147 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022703 4965 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:12:22.037636 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022708 4965 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:12:22.037636 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022712 4965 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:12:22.037636 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022717 4965 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:12:22.037636 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022721 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:12:22.037636 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022725 4965 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:12:22.037636 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022730 4965 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:12:22.037636 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022735 4965 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:12:22.037636 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022739 4965 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:12:22.037636 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022743 4965 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:12:22.037636 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022748 4965 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:12:22.037636 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022752 4965 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:12:22.037636 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022756 4965 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:12:22.037636 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022761 4965 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:12:22.037636 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022765 4965 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:12:22.037636 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022769 4965 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:12:22.037636 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022773 4965 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:12:22.037636 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022778 4965 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:12:22.037636 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022782 4965 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:12:22.037636 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022787 4965 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:12:22.038124 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022791 4965 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:12:22.038124 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022795 4965 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:12:22.038124 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022800 4965 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:12:22.038124 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022804 4965 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:12:22.038124 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022810 4965 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:12:22.038124 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.022816 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:12:22.038124 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.022874 4965 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:12:22.038124 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023083 4965 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:12:22.038124 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023091 4965 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:12:22.038124 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023096 4965 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:12:22.038124 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023100 4965 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:12:22.038124 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023104 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:12:22.038124 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023109 4965 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:12:22.038124 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023113 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:12:22.038124 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023118 4965 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:12:22.038124 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023122 4965 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:12:22.038560 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023126 4965 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:12:22.038560 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023131 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:12:22.038560 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023135 4965 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:12:22.038560 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023140 4965 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:12:22.038560 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023145 4965 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:12:22.038560 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023149 4965 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:12:22.038560 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023153 4965 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:12:22.038560 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023158 4965 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:12:22.038560 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023162 4965 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:12:22.038560 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023167 4965 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:12:22.038560 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023171 4965 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:12:22.038560 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023175 4965 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:12:22.038560 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023179 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:12:22.038560 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023184 4965 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:12:22.038560 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023188 4965 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:12:22.038560 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023192 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:12:22.038560 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023197 4965 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:12:22.038560 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023203 4965 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:12:22.038560 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023210 4965 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:12:22.039052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023216 4965 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:12:22.039052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023223 4965 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:12:22.039052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023228 4965 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:12:22.039052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023233 4965 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:12:22.039052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023238 4965 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:12:22.039052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023242 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:12:22.039052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023247 4965 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:12:22.039052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023251 4965 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:12:22.039052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023256 4965 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:12:22.039052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023260 4965 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:12:22.039052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023264 4965 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:12:22.039052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023268 4965 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:12:22.039052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023273 4965 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:12:22.039052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023278 4965 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:12:22.039052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023282 4965 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:12:22.039052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023286 4965 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:12:22.039052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023290 4965 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:12:22.039052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023295 4965 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:12:22.039052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023300 4965 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:12:22.039505 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023304 4965 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:12:22.039505 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023308 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:12:22.039505 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023312 4965 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:12:22.039505 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023317 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:12:22.039505 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023322 4965 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:12:22.039505 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023326 4965 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:12:22.039505 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023405 4965 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:12:22.039505 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023483 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:12:22.039505 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023520 4965 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:12:22.039505 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023527 4965 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:12:22.039505 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023534 4965 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:12:22.039505 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023540 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:12:22.039505 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023545 4965 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:12:22.039505 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023549 4965 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:12:22.039505 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023554 4965 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:12:22.039505 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023559 4965 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:12:22.039505 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023564 4965 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:12:22.039505 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023568 4965 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:12:22.039505 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023573 4965 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:12:22.039505 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023577 4965 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:12:22.040115 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023588 4965 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:12:22.040115 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023593 4965 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:12:22.040115 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023597 4965 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:12:22.040115 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023602 4965 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:12:22.040115 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023606 4965 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:12:22.040115 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023611 4965 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:12:22.040115 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023615 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:12:22.040115 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023622 4965 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:12:22.040115 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023627 4965 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:12:22.040115 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023631 4965 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:12:22.040115 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023636 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:12:22.040115 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023641 4965 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:12:22.040115 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023651 4965 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:12:22.040115 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023656 4965 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:12:22.040115 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023661 4965 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:12:22.040115 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023665 4965 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:12:22.040115 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023669 4965 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:12:22.040115 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023674 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:12:22.040115 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023678 4965 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:12:22.040604 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023684 4965 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:12:22.040604 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023688 4965 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:12:22.040604 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023694 4965 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:12:22.040604 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023699 4965 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:12:22.040604 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023703 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:12:22.040604 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023707 4965 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:12:22.040604 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023716 4965 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:12:22.040604 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023720 4965 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:12:22.040604 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023725 4965 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:12:22.040604 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023729 4965 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:12:22.040604 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023733 4965 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:12:22.040604 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023738 4965 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:12:22.040604 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023742 4965 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:12:22.040604 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023747 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:12:22.040604 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023751 4965 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:12:22.040604 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023755 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:12:22.040604 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023759 4965 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:12:22.040604 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023764 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:12:22.040604 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023768 4965 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:12:22.040604 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023778 4965 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:12:22.041125 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.023835 4965 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:12:22.041125 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023986 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:12:22.041125 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.023998 4965 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:12:22.041125 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024004 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:12:22.041125 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024009 4965 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:12:22.041125 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024014 4965 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:12:22.041125 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024019 4965 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:12:22.041125 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024023 4965 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:12:22.041125 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024028 4965 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:12:22.041125 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024032 4965 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:12:22.041125 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024036 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:12:22.041125 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024042 4965 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:12:22.041125 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024049 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:12:22.041125 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024058 4965 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:12:22.041125 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024063 4965 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:12:22.041125 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024068 4965 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:12:22.041523 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024073 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:12:22.041523 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024077 4965 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:12:22.041523 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024081 4965 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:12:22.041523 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024085 4965 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:12:22.041523 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024089 4965 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:12:22.041523 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024093 4965 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:12:22.041523 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024098 4965 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:12:22.041523 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024102 4965 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:12:22.041523 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024106 4965 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:12:22.041523 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024111 4965 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:12:22.041523 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024120 4965 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:12:22.041523 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024124 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:12:22.041523 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024128 4965 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:12:22.041523 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024133 4965 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:12:22.041523 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024137 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:12:22.041523 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024141 4965 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:12:22.041523 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024148 4965 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:12:22.041523 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024156 4965 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:12:22.041523 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024164 4965 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:12:22.042052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024169 4965 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:12:22.042052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024174 4965 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:12:22.042052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024179 4965 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:12:22.042052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024188 4965 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:12:22.042052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024192 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:12:22.042052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024197 4965 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:12:22.042052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024201 4965 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:12:22.042052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024206 4965 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:12:22.042052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024211 4965 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:12:22.042052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024217 4965 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:12:22.042052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024222 4965 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:12:22.042052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024226 4965 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:12:22.042052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024231 4965 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:12:22.042052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024236 4965 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:12:22.042052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024242 4965 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:12:22.042052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024252 4965 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:12:22.042052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024256 4965 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:12:22.042052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024260 4965 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:12:22.042052 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024264 4965 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:12:22.042527 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024269 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:12:22.042527 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024273 4965 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:12:22.042527 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024277 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:12:22.042527 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024282 4965 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:12:22.042527 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024287 4965 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:12:22.042527 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024291 4965 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:12:22.042527 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024295 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:12:22.042527 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024299 4965 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:12:22.042527 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024308 4965 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:12:22.042527 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024312 4965 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:12:22.042527 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024316 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:12:22.042527 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024320 4965 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:12:22.042527 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024327 4965 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:12:22.042527 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024331 4965 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:12:22.042527 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024335 4965 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:12:22.042527 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024340 4965 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:12:22.042527 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024344 4965 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:12:22.042527 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024348 4965 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:12:22.042527 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024352 4965 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:12:22.042527 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024361 4965 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:12:22.043157 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024375 4965 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:12:22.043157 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024381 4965 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:12:22.043157 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024386 4965 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:12:22.043157 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024392 4965 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:12:22.043157 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024398 4965 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:12:22.043157 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024403 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:12:22.043157 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024428 4965 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:12:22.043157 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024449 4965 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:12:22.043157 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024453 4965 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:12:22.043157 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024456 4965 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:12:22.043157 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024459 4965 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:12:22.043157 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024561 4965 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:12:22.043157 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024565 4965 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:12:22.043157 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024568 4965 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:12:22.043157 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024571 4965 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:12:22.043157 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024574 4965 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:12:22.043157 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024576 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:12:22.043157 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024579 4965 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:12:22.043157 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024582 4965 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:12:22.043623 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024585 4965 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:12:22.043623 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024589 4965 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:12:22.043623 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024591 4965 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:12:22.043623 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024594 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:12:22.043623 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024597 4965 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:12:22.043623 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024600 4965 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:12:22.043623 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024603 4965 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:12:22.043623 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024607 4965 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:12:22.043623 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024612 4965 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:12:22.043623 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024616 4965 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:12:22.043623 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024620 4965 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:12:22.043623 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024629 4965 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:12:22.043623 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024633 4965 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:12:22.043623 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:22.024637 4965 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:12:22.043623 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.024723 4965 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 17:12:22.043623 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.025837 4965 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 17:12:22.043623 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.025918 4965 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 17:12:22.043623 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.026115 4965 server.go:1014] "Starting client certificate rotation" Oct 30 17:12:22.043623 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.026254 4965 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 17:12:22.044384 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.026321 4965 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 17:06:41 +0000 UTC" deadline="2028-05-27 11:02:33.190357375 +0000 UTC" Oct 30 17:12:22.044384 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.026359 4965 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13793h50m11.164001493s" Oct 30 17:12:22.044384 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.026583 4965 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:12:22.044384 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.027660 4965 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:12:22.044384 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.028975 4965 log.go:25] "Validated CRI v1 runtime API" Oct 30 17:12:22.044384 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.030714 4965 log.go:25] "Validated CRI v1 image API" Oct 30 17:12:22.044384 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.031483 4965 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 17:12:22.044384 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.033449 4965 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8f1535e6-2162-42c8-a77d-ce4389de7ebe:/dev/nvme0n1p3 bf976311-5ed8-4095-90b2-5bbf64d61b63:/dev/nvme0n1p4] Oct 30 17:12:22.044788 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.033466 4965 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1022106440a5e601a7e59d8d4ad406e94fd0f7fed1b15d885e20f8c70481c57e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1022106440a5e601a7e59d8d4ad406e94fd0f7fed1b15d885e20f8c70481c57e/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/128cefb7c44adc89580c33e6ad06c84105963070eebc6e81f53281534582965b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/128cefb7c44adc89580c33e6ad06c84105963070eebc6e81f53281534582965b/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f5fa8d664b221e5e8391e925d46ce9b0fa0c0e8061496e0a6ffb5aaad6138a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f5fa8d664b221e5e8391e925d46ce9b0fa0c0e8061496e0a6ffb5aaad6138a7/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/43de827525a68f6af4bfe173cd354ea23de185a40e5eda53bba8309420b7571c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/43de827525a68f6af4bfe173cd354ea23de185a40e5eda53bba8309420b7571c/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4d7da57858bd0907a716dbaac42f1783b0ace83acd5f89ab37219848d1b5d0d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4d7da57858bd0907a716dbaac42f1783b0ace83acd5f89ab37219848d1b5d0d9/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5cc65189d640b892290d7cf378248b8fbaeab9d260eac05bb6527647b0fed825/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5cc65189d640b892290d7cf378248b8fbaeab9d260eac05bb6527647b0fed825/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/69639bd161d7b5158a9ebe9374202d313f6c64a12bfc34b5ebe75a9030a5a215/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/69639bd161d7b5158a9ebe9374202d313f6c64a12bfc34b5ebe75a9030a5a215/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b5f03698ad2b8d27127136fb8ac9b2d5f1f6cfdfdc2a1cc7c2e7b8540ee4663/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b5f03698ad2b8d27127136fb8ac9b2d5f1f6cfdfdc2a1cc7c2e7b8540ee4663/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/74f894480aa0bf718f1fbba10d211a7853089a2b067b1e45fe10b6c2f05f6094/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/74f894480aa0bf718f1fbba10d211a7853089a2b067b1e45fe10b6c2f05f6094/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/837040638b1b42bf20445cb551dc04c6277f9fbef3a55987538fc7f0b9bf8c15/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/837040638b1b42bf20445cb551dc04c6277f9fbef3a55987538fc7f0b9bf8c15/userdata/shm major:0 minor:328 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8860d3615245d0a04f931cc48968445461dce11d162d2a8665fd127bddd7c6ae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8860d3615245d0a04f931cc48968445461dce11d162d2a8665fd127bddd7c6ae/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8956acc30b6dcb37c2361fa375236376b6d9aeb48ebe8359497860c622250a7a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8956acc30b6dcb37c2361fa375236376b6d9aeb48ebe8359497860c622250a7a/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a9cf72d49e9c9eec7405103c22c662041354712ebc826ea78866070fbc7aff8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a9cf72d49e9c9eec7405103c22c662041354712ebc826ea78866070fbc7aff8/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8f7b8781d9e26722564e26832573d120947b1c44750561ef7848fa85b315d8e9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8f7b8781d9e26722564e26832573d120947b1c44750561ef7848fa85b315d8e9/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9428e795c347bf480a762ae94fca6bd2f8e9a1629cb722c587615ae8ef671124/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9428e795c347bf480a762ae94fca6bd2f8e9a1629cb722c587615ae8ef671124/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a41caba616cb22e3dc315b213de1e63cca96a70c659fb76d595aaa1273494dc0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a41caba616cb22e3dc315b213de1e63cca96a70c659fb76d595aaa1273494dc0/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1f4230704e571ae81b3c38071fcfbdcdf9ff5933438e5bdbb3e99520e76bcee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1f4230704e571ae81b3c38071fcfbdcdf9ff5933438e5bdbb3e99520e76bcee/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b333be98afc85046a05ce164d47c64803e19f7598f1591771baf6ad2e71317d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b333be98afc85046a05ce164d47c64803e19f7598f1591771baf6ad2e71317d8/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/caa5716795758214e8a79342a54eedf4ec7ced420359018bbd53847e75eb49a3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/caa5716795758214e8a79342a54eedf4ec7ced420359018bbd53847e75eb49a3/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d46a7f3e70d669027e4930adb828ba01c2c6e5c44f0a5bf10cdc0d06f1e10b38/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d46a7f3e70d669027e4930adb828ba01c2c6e5c44f0a5bf10cdc0d06f1e10b38/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d76f4b44b8637a5ec189760f43d92a4b83070de536bf400ee08fdc21700fd49a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d76f4b44b8637a5ec189760f43d92a4b83070de536bf400ee08fdc21700fd49a/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e8a68f579b2467a036f5512235b4620091a569e46370b57a92317184447bc764/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e8a68f579b2467a036f5512235b4620091a569e46370b57a92317184447bc764/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eefca769132f66ee6473ad8447eb6a35ff9c5c8c8b82fbccf7aaf883425b9373/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eefca769132f66ee6473ad8447eb6a35ff9c5c8c8b82fbccf7aaf883425b9373/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f585d4b8a2f2b3f0fe64149777c82cc42afa2e12c13f13b201f0d616827c6251/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f585d4b8a2f2b3f0fe64149777c82cc42afa2e12c13f13b201f0d616827c6251/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03a0e7f9-9e86-4aa9-9864-205263f6a486/volumes/kubernetes.io~projected/kube-api-access-txhrv:{mountpoint:/var/lib/kubelet/pods/03a0e7f9-9e86-4aa9-9864-205263f6a486/volumes/kubernetes.io~projected/kube-api-access-txhrv major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13444fae-8e71-41ee-8489-55b8606492fb/volumes/kubernetes.io~projected/kube-api-access-drkzm:{mountpoint:/var/lib/kubelet/pods/13444fae-8e71-41ee-8489-55b8606492fb/volumes/kubernetes.io~projected/kube-api-access-drkzm major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16cd937e-4c6d-45d0-a9de-f77dc836d082/volumes/kubernetes.io~projected/kube-api-access-wtlv4:{mountpoint:/var/lib/kubelet/pods/16cd937e-4c6d-45d0-a9de-f77dc836d082/volumes/kubernetes.io~projected/kube-api-access-wtlv4 major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16cd937e-4c6d-45d0-a9de-f77dc836d082/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/16cd937e-4c6d-45d0-a9de-f77dc836d082/volumes/kubernetes.io~secret/serving-cert major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/195f952d-ee96-485d-99aa-8b47e6633c97/volumes/kubernetes.io~projected/kube-api-access-l25lh:{mountpoint:/var/lib/kubelet/pods/195f952d-ee96-485d-99aa-8b47e6633c97/volumes/kubernetes.io~projected/kube-api-access-l25lh major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/195f952d-ee96-485d-99aa-8b47e6633c97/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/195f952d-ee96-485d-99aa-8b47e6633c97/volumes/kubernetes.io~secret/serving-cert major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1eab122e-07da-4307-b7f2-914cbe2b4237/volumes/kubernetes.io~projected/kube-api-access-f9kmm:{mountpoint:/var/lib/kubelet/pods/1eab122e-07da-4307-b7f2-914cbe2b4237/volumes/kubernetes.io~projected/kube-api-access-f9kmm major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1eab122e-07da-4307-b7f2-914cbe2b4237/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/1eab122e-07da-4307-b7f2-914cbe2b4237/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2664a881-44d7-470a-93d0-df1b3eaca0bc/volumes/kubernetes.io~projected/kube-api-access-p5mmm:{mountpoint:/var/lib/kubelet/pods/2664a881-44d7-470a-93d0-df1b3eaca0bc/volumes/kubernetes.io~projected/kube-api-access-p5mmm major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27ac623c-2f57-44ec-a1b1-3ad87909452d/volumes/kubernetes.io~projected/kube-api-access-g4qm2:{mountpoint:/var/lib/kubelet/pods/27ac623c-2f57-44ec-a1b1-3ad87909452d/volumes/kubernetes.io~projected/kube-api-access-g4qm2 major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27ac623c-2f57-44ec-a1b1-3ad87909452d/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/27ac623c-2f57-44ec-a1b1-3ad87909452d/volumes/kubernetes.io~secret/serving-cert major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2906e95e-3baa-4c79-ad9c-b2592c1d414f/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/2906e95e-3baa-4c79-ad9c-b2592c1d414f/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2939b3a2-1950-44c9-802c-a53861aa7aa8/volumes/kubernetes.io~projected/kube-api-access-ffnbr:{mountpoint:/var/lib/kubelet/pods/2939b3a2-1950-44c9-802c-a53861aa7aa8/volumes/kubernetes.io~projected/kube-api-access-ffnbr major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2989c0bb-0ed6-41b2-ad3e-bb6e85bd60b9/volumes/kubernetes.io~projected/kube-api-access-csqbr:{mountpoint:/var/lib/kubelet/pods/2989c0bb-0ed6-41b2-ad3e-bb6e85bd60b9/volumes/kubernetes.io~projected/kube-api-access-csqbr major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/467187ee-7039-4bb4-adbe-38161ee7c336/volumes/kubernetes.io~projected/kube-api-access-p7n64:{mountpoint:/var/lib/kubelet/pods/467187ee-7039-4bb4-adbe-38161ee7c336/volumes/kubernetes.io~projected/kube-api-access-p7n64 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4bdc401b-ae3f-48c0-8110-8cd6096007d8/volumes/kubernetes.io~projected/kube-api-access-xkvxq:{mountpoint:/var/lib/kubelet/pods/4bdc401b-ae3f-48c0-8110-8cd6096007d8/volumes/kubernetes.io~projected/kube-api-access-xkvxq major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4bdc401b-ae3f-48c0-8110-8cd6096007d8/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/4bdc401b-ae3f-48c0-8110-8cd6096007d8/volumes/kubernetes.io~secret/signing-key major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55f62602-c7b7-4a6c-b244-f6e48775733f/volumes/kubernetes.io~projected/kube-api-access-m9swv:{mountpoint:/var/lib/kubelet/pods/55f62602-c7b7-4a6c-b244-f6e48775733f/volumes/kubernetes.io~projected/kube-api-access-m9swv major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6084e9b4-6b79-46ab-91cf-8d292d1f4591/volumes/kubernetes.io~projected/kube-api-access-9zxfh:{mountpoint:/var/lib/kubelet/pods/6084e9b4-6b79-46ab-91cf-8d292d1f4591/volumes/kubernetes.io~projected/kube-api-access-9zxfh major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6178534d-9f75-49c7-a977-de65737cbcea/volumes/kubernetes.io~projected/kube-api-access-nqrt8:{mountpoint:/var/lib/kubelet/pods/6178534d-9f75-49c7-a977-de65737cbcea/volumes/kubernetes.io~projected/kube-api-access-nqrt8 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/745f4f0f-74f7-4fff-9384-c8f45f601e33/volumes/kubernetes.io~projected/kube-api-access-ztbbl:{mountpoint:/var/lib/kubelet/pods/745f4f0f-74f7-4fff-9384-c8f45f601e33/volumes/kubernetes.io~projected/kube-api-access-ztbbl major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89eff971-c01b-4b06-9c16-4332956558a3/volumes/kubernetes.io~projected/kube-api-access-ddss9:{mountpoint:/var/lib/kubelet/pods/89eff971-c01b-4b06-9c16-4332956558a3/volumes/kubernetes.io~projected/kube-api-access-ddss9 major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89eff971-c01b-4b06-9c16-4332956558a3/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/89eff971-c01b-4b06-9c16-4332956558a3/volumes/kubernetes.io~secret/cert major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/924ac066-ff98-4320-9eb2-268f0a39ca33/volumes/kubernetes.io~projected/kube-api-access-mrqrz:{mountpoint:/var/lib/kubelet/pods/924ac066-ff98-4320-9eb2-268f0a39ca33/volumes/kubernetes.io~projected/kube-api-access-mrqrz major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/924ac066-ff98-4320-9eb2-268f0a39ca33/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/924ac066-ff98-4320-9eb2-268f0a39ca33/volumes/kubernetes.io~secret/default-certificate major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/924ac066-ff98-4320-9eb2-268f0a39ca33/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/924ac066-ff98-4320-9eb2-268f0a39ca33/volumes/kubernetes.io~secret/stats-auth major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/933be7f4-8d62-4735-807f-7aa1439583fa/volumes/kubernetes.io~projected/kube-api-access-6sh5q:{mountpoint:/var/lib/kubelet/pods/933be7f4-8d62-4735-807f-7aa1439583fa/volumes/kubernetes.io~projected/kube-api-access-6sh5q major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce/volumes/kubernetes.io~projected/kube-api-access-brgv8:{mountpoint:/var/lib/kubelet/pods/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce/volumes/kubernetes.io~projected/kube-api-access-brgv8 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7ac81c7-e1f0-4549-886d-d7b8b893b614/volumes/kubernetes.io~projected/kube-api-access-5kcdc:{mountpoint:/var/lib/kubelet/pods/c7ac81c7-e1f0-4549-886d-d7b8b893b614/volumes/kubernetes.io~projected/kube-api-access-5kcdc major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7b7edd7-11cc-4fac-92ae-a08140056adb/volumes/kubernetes.io~projected/kube-api-access-xs2g4:{mountpoint:/var/lib/kubelet/pods/c7b7edd7-11cc-4fac-92ae-a08140056adb/volumes/kubernetes.io~projected/kube-api-access-xs2g4 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc71466d-eac0-460b-af50-3204f07b3f35/volumes/kubernetes.io~projected/kube-api-access-rttdq:{mountpoint:/var/lib/kubelet/pods/cc71466d-eac0-460b-af50-3204f07b3f35/volumes/kubernetes.io~projected/kube-api-access-rttdq major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd6ddeda-aec9-4872-9e43-084577cfcd7d/volumes/kubernetes.io~projected/kube-api-access-r67qt:{mountpoint:/var/lib/kubelet/pods/cd6ddeda-aec9-4872-9e43-084577cfcd7d/volumes/kubernetes.io~projected/kube-api-access-r67qt major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d05e04a8-3ddf-4261-add0-c2324c6c9e21/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/d05e04a8-3ddf-4261-add0-c2324c6c9e21/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d05e04a8-3ddf-4261-add0-c2324c6c9e21/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/d05e04a8-3ddf-4261-add0-c2324c6c9e21/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d05e04a8-3ddf-4261-add0-c2324c6c9e21/volumes/kubernetes.io~projected/kube-api-access-jn8lc:{mountpoint:/var/lib/kubelet/pods/d05e04a8-3ddf-4261-add0-c2324c6c9e21/volumes/kubernetes.io~projected/kube-api-access-jn8lc major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d39ddd3a-781f-4084-b4e2-624b130c1025/volumes/kubernetes.io~projected/kube-api-access-z4njs:{mountpoint:/var/lib/kubelet/pods/d39ddd3a-781f-4084-b4e2-624b130c1025/volumes/kubernetes.io~projected/kube-api-access-z4njs major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e30e6b59-b8e2-4125-ae7a-41edd1ce7389/volumes/kubernetes.io~projected/kube-api-access-5dz8c:{mountpoint:/var/lib/kubelet/pods/e30e6b59-b8e2-4125-ae7a-41edd1ce7389/volumes/kubernetes.io~projected/kube-api-access-5dz8c major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5f38da3-fe1f-4f36-8ed3-12cb01bf128b/volumes/kubernetes.io~projected/kube-api-access-5zz48:{mountpoint:/var/lib/kubelet/pods/e5f38da3-fe1f-4f36-8ed3-12cb01bf128b/volumes/kubernetes.io~projected/kube-api-access-5zz48 major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f442e60f-8ccf-485e-9709-60a909d482bc/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/f442e60f-8ccf-485e-9709-60a909d482bc/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f608efeb-1bda-4375-a428-c6db6341fd0b/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f608efeb-1bda-4375-a428-c6db6341fd0b/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f608efeb-1bda-4375-a428-c6db6341fd0b/volumes/kubernetes.io~projected/kube-api-access-d49vx:{mountpoint:/var/lib/kubelet/pods/f608efeb-1bda-4375-a428-c6db6341fd0b/volumes/kubernetes.io~projected/kube-api-access-d49vx major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f608efeb-1bda-4375-a428-c6db6341fd0b/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f608efeb-1bda-4375-a428-c6db6341fd0b/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd74ba94-86d0-4707-94e5-6822c4ab5751/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/fd74ba94-86d0-4707-94e5-6822c4ab5751/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd74ba94-86d0-4707-94e5-6822c4ab5751/volumes/kubernetes.io~projected/kube-api-access-qx4rg:{mountpoint:/var/lib/kubelet/pods/fd74ba94-86d0-4707-94e5-6822c4ab5751/volumes/kubernetes.io~projected/kube-api-access-qx4rg major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd74ba94-86d0-4707-94e5-6822c4ab5751/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/fd74ba94-86d0-4707-94e5-6822c4ab5751/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd74ba94-86d0-4707-94e5-6822c4ab5751/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/fd74ba94-86d0-4707-94e5-6822c4ab5751/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:215 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/f4cd97668f95028355a0b75cf18116aa21512b5d786a24c152eaa385937f1b7b/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/687d905cc7350c1739e793741756ab6df6058835ba69eb70c1917eca93f557aa/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/6f879e81f6ae26d6dcdcf6983e4db47e34a1e67b67383cb25a8bb89b8246e243/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/08b4bdc6b90a10804c8a710012ca4f7551d37696a34a19edd018609f27c4f04e/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/c6463e3c580583421ba8e228382d91040180b9dfc8215c77e2bca18eb6bcc952/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/0f73e54a7c56a66d98ec713e5df6f59bb623561c920d79d1d8da9503d93da697/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/9428bfbaf8221d6fa35f3215df4413a9d3b5b94589cff46589e9c24e5459f00f/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/56cda86ff47429fdb4da9fc774021402895f3217b744d9e64ad9d3b4fab0df82/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/21d43d96110bb2d63646c6f59d8d2e4d381e9274cf735e06aecf525553c64520/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/d77e930cd5b0896527ddd4b5c878e7499d07f42eb2f74ddd54028a878a0bc9b0/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/7ff316beccd5d5688ea417901fc9e173fb408544b1dc2ff6008de8932751ea74/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/80649f9c6285f387e435e81c75e615b30f2a70fa7357007c11a9e9cf8ba9be34/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/f86a1127935e7ae5c5806aadde8a0f30895bbadf8735d0a7b95b4a4edb112ac8/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/0fb313bcb6ffb5a898bca0d5666d2fdece7f926a43fcb300088caecb5a53ccff/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/da01abfd20c2683616ded63952b81732378e0b3107b2e8574eeaa28ac8a169d0/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/e87455ebde3734ea73f18912967a76b3caa52374d837b079e7f612d7231eb800/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/86e3c4b28d280145baf976031ace63417ba5ade33e11d935c79b0e88d4c2a81f/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/5c5b0e2366137d4b7b95cfb4c93ef3fc644d2473727f02aa11e8210e11738a87/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/5445d69bc867cc6246fe160b8147a72cddd2a3027ee027e6b126485cd6ea0a36/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/81cea87dcc2d537eb25e6d27470b76279fb26ef3638e27d11b1455b8066bf4ff/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/5bda0a91036adfe713ee0efe6c02534dbbfe4c1354ca95a8f8c5c6c536661833/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/c04097d66520dd055b06cb204b3355b6c1d92dd4e14a0cb77f7963a94f8580e4/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/8759d7aa13ca0976133c36f95346875aeb2dbc3dfcd781c2f06ebecce544af57/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/84926db13ec1e7a4b241f16ba130ddbee5d230ff80287eb5f21f24195aefa2e7/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/fe17d728206dd53a4bfa4a0616577cbc687d1bbb61f825ef73d89a3c2584722f/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/56cb0fe91c7fa68119c2d18c00f5267c171d60dfc2479b5671b035fb2783de3c/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/922e49453882f9852f19560c1911b88f0819b782cdfb5b6684edea172a4f38d5/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/0b44fa5908b4d15db435e6ba68afe2eb29d155d39f4c945447e9fe6c7c1e84ff/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/cb682c025172b1b0d50f3d650440a1120163923e6ff53f9d7ad06abe71f3d787/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/6bde96eadba37e4d69103cb07573d46939b5959671cc3910d88a469e2c99f8b3/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/4cb9ee1d1bef1bbcf0a4df6588095aed27df11c98c3ccc93478d71df4ca0d017/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/bd801a9238dc96714d24a2a775f6beea805fd23a51fc2f37575080c19761c45f/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/37aad00cf52cc27178edc47323583b0eda4391cb3f5a135efb288bcda6a24bfb/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/5abcd1a03685b232a06b827cf1c1927e6c38a2fc8ec7aaa959924f56b91ca047/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/e90369d6acb46bc931cfb145df5a74a9600c3966a7e8129f9142de38fcdc2f28/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/45d0b4d37b684d6a66f927b26449fbb240deb2d5245b26a5cc7af96a1631de8c/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/ecff5946f9a1a72500fd84c6d3fa894744646917548269139e7e61a147e0329f/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/677866f8d5070be378b4175ba45d5e2f635248f8458345114c3ee7fdfb18bb80/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/4b04d5a3fe65c865f522b6f5ec47166a5a752bfbfc6278a7369eaccc319a8f37/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/c76e534e1574fca2370c229ff4ec90f64d179ad791c747d7b0dfb27b72c1069b/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/71ff45757f912eb2b7c229c0377d100a56b797a6968035037b90d4ad752ea998/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/cff53bd1df9c3ad740efa8463c73addaf58047a5f450dc0ca487b142acab9ac9/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/2ae1fdf044a12c1ee357b40fad02d223f42193c35db33b234a995b8a4f1f1004/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/ff910e27f704f68b67c0bd8757701de37363bb2bd7c5e696250b619495607c33/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/38e6d4bff637090dce47f3a5c44b11e762d41f721bbaf202e05c8ea9a72c7bce/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/7bee0f63ac3839b73b5cb290199ad9122f13c64371bd2a025cc043af538670f4/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/9ae2c5c2c2e6f76131c7992851fac3c6ab75a994c63c81a7b3b0294f2f5a5e26/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/0283670105b5c92ad82bf71fac0c39290809a960a6d28245b4492cc964d1cf43/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/a39728cbb9a0a84d7edde359b578d07744cad617cc845b556e44b4d868917ca5/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/a8afc55f1da28f821abf5b3ab64b0d4e6353af99322ddaea58faf8b5946cf6fe/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/d00ab0a07ab05eef1341b03d272944f228ffea8edb7d2bf439211fa934db8886/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/d88bb88cc848b8b091c2649e5572d1af9667b0f35c16ebfbe0a2588600b0823a/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/c257783fde4642f67e1f91962e188b457e51b17f670bbe772bcab8500ac58bf8/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/3b81f4729db1205f1b9b7b1138e14bd26df87214b71aa58bd466fca40e02988c/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/729a93f0c763e4a3f8799099e3c585d2d05417ebb98cc0d881c0d24dda28b996/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/ea39fed43c0d6f6d920893fe96f2223aaa116bae96cbe4a52e7f2d8f08e32200/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/e7d79ab649662263e479ed216b01ce8ee767d36251db478890868f2f1aacba3c/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/1699928f7f951c308e87124bc30d68e73b82cd6315e253b3b6be2af4114faaa5/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 17:12:22.045187 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.042471 4965 manager.go:217] Machine: {Timestamp:2026-10-30 17:12:22.04198984 +0000 UTC m=+0.094089763 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099766 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f0b6ac46ee1dc4d12f791bade63f8 SystemUUID:ec2f0b6a-c46e-e1dc-4d12-f791bade63f8 BootID:1a2be75d-e57e-424b-9ebb-9086aa8444ae Filesystems:[{Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce/volumes/kubernetes.io~projected/kube-api-access-brgv8 DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b5f03698ad2b8d27127136fb8ac9b2d5f1f6cfdfdc2a1cc7c2e7b8540ee4663/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f608efeb-1bda-4375-a428-c6db6341fd0b/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/27ac623c-2f57-44ec-a1b1-3ad87909452d/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:230 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/89eff971-c01b-4b06-9c16-4332956558a3/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:231 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1eab122e-07da-4307-b7f2-914cbe2b4237/volumes/kubernetes.io~projected/kube-api-access-f9kmm DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e5f38da3-fe1f-4f36-8ed3-12cb01bf128b/volumes/kubernetes.io~projected/kube-api-access-5zz48 DeviceMajor:0 DeviceMinor:232 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9428e795c347bf480a762ae94fca6bd2f8e9a1629cb722c587615ae8ef671124/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d39ddd3a-781f-4084-b4e2-624b130c1025/volumes/kubernetes.io~projected/kube-api-access-z4njs DeviceMajor:0 DeviceMinor:293 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e30e6b59-b8e2-4125-ae7a-41edd1ce7389/volumes/kubernetes.io~projected/kube-api-access-5dz8c DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/933be7f4-8d62-4735-807f-7aa1439583fa/volumes/kubernetes.io~projected/kube-api-access-6sh5q DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1022106440a5e601a7e59d8d4ad406e94fd0f7fed1b15d885e20f8c70481c57e/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2989c0bb-0ed6-41b2-ad3e-bb6e85bd60b9/volumes/kubernetes.io~projected/kube-api-access-csqbr DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/195f952d-ee96-485d-99aa-8b47e6633c97/volumes/kubernetes.io~projected/kube-api-access-l25lh DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2664a881-44d7-470a-93d0-df1b3eaca0bc/volumes/kubernetes.io~projected/kube-api-access-p5mmm DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/13444fae-8e71-41ee-8489-55b8606492fb/volumes/kubernetes.io~projected/kube-api-access-drkzm DeviceMajor:0 DeviceMinor:323 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/55f62602-c7b7-4a6c-b244-f6e48775733f/volumes/kubernetes.io~projected/kube-api-access-m9swv DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/195f952d-ee96-485d-99aa-8b47e6633c97/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/27ac623c-2f57-44ec-a1b1-3ad87909452d/volumes/kubernetes.io~projected/kube-api-access-g4qm2 DeviceMajor:0 DeviceMinor:237 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fd74ba94-86d0-4707-94e5-6822c4ab5751/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/89eff971-c01b-4b06-9c16-4332956558a3/volumes/kubernetes.io~projected/kube-api-access-ddss9 DeviceMajor:0 DeviceMinor:233 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/74f894480aa0bf718f1fbba10d211a7853089a2b067b1e45fe10b6c2f05f6094/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1f4230704e571ae81b3c38071fcfbdcdf9ff5933438e5bdbb3e99520e76bcee/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6178534d-9f75-49c7-a977-de65737cbcea/volumes/kubernetes.io~projected/kube-api-access-nqrt8 DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a9cf72d49e9c9eec7405103c22c662041354712ebc826ea78866070fbc7aff8/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d05e04a8-3ddf-4261-add0-c2324c6c9e21/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cd6ddeda-aec9-4872-9e43-084577cfcd7d/volumes/kubernetes.io~projected/kube-api-access-r67qt DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d76f4b44b8637a5ec189760f43d92a4b83070de536bf400ee08fdc21700fd49a/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/128cefb7c44adc89580c33e6ad06c84105963070eebc6e81f53281534582965b/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f5fa8d664b221e5e8391e925d46ce9b0fa0c0e8061496e0a6ffb5aaad6138a7/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/745f4f0f-74f7-4fff-9384-c8f45f601e33/volumes/kubernetes.io~projected/kube-api-access-ztbbl DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2939b3a2-1950-44c9-802c-a53861aa7aa8/volumes/kubernetes.io~projected/kube-api-access-ffnbr DeviceMajor:0 DeviceMinor:235 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b333be98afc85046a05ce164d47c64803e19f7598f1591771baf6ad2e71317d8/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f585d4b8a2f2b3f0fe64149777c82cc42afa2e12c13f13b201f0d616827c6251/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/16cd937e-4c6d-45d0-a9de-f77dc836d082/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cc71466d-eac0-460b-af50-3204f07b3f35/volumes/kubernetes.io~projected/kube-api-access-rttdq DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4bdc401b-ae3f-48c0-8110-8cd6096007d8/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:336 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/f608efeb-1bda-4375-a428-c6db6341fd0b/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/924ac066-ff98-4320-9eb2-268f0a39ca33/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/f442e60f-8ccf-485e-9709-60a909d482bc/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d05e04a8-3ddf-4261-add0-c2324c6c9e21/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8956acc30b6dcb37c2361fa375236376b6d9aeb48ebe8359497860c622250a7a/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/16cd937e-4c6d-45d0-a9de-f77dc836d082/volumes/kubernetes.io~projected/kube-api-access-wtlv4 DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4bdc401b-ae3f-48c0-8110-8cd6096007d8/volumes/kubernetes.io~projected/kube-api-access-xkvxq DeviceMajor:0 DeviceMinor:337 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7b7edd7-11cc-4fac-92ae-a08140056adb/volumes/kubernetes.io~projected/kube-api-access-xs2g4 DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/03a0e7f9-9e86-4aa9-9864-205263f6a486/volumes/kubernetes.io~projected/kube-api-access-txhrv DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eefca769132f66ee6473ad8447eb6a35ff9c5c8c8b82fbccf7aaf883425b9373/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fd74ba94-86d0-4707-94e5-6822c4ab5751/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/924ac066-ff98-4320-9eb2-268f0a39ca33/volumes/kubernetes.io~projected/kube-api-access-mrqrz DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/43de827525a68f6af4bfe173cd354ea23de185a40e5eda53bba8309420b7571c/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f608efeb-1bda-4375-a428-c6db6341fd0b/volumes/kubernetes.io~projected/kube-api-access-d49vx DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/467187ee-7039-4bb4-adbe-38161ee7c336/volumes/kubernetes.io~projected/kube-api-access-p7n64 DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/69639bd161d7b5158a9ebe9374202d313f6c64a12bfc34b5ebe75a9030a5a215/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2906e95e-3baa-4c79-ad9c-b2592c1d414f/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6084e9b4-6b79-46ab-91cf-8d292d1f4591/volumes/kubernetes.io~projected/kube-api-access-9zxfh DeviceMajor:0 DeviceMinor:316 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d46a7f3e70d669027e4930adb828ba01c2c6e5c44f0a5bf10cdc0d06f1e10b38/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e8a68f579b2467a036f5512235b4620091a569e46370b57a92317184447bc764/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/837040638b1b42bf20445cb551dc04c6277f9fbef3a55987538fc7f0b9bf8c15/userdata/shm DeviceMajor:0 DeviceMinor:328 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a41caba616cb22e3dc315b213de1e63cca96a70c659fb76d595aaa1273494dc0/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fd74ba94-86d0-4707-94e5-6822c4ab5751/volumes/kubernetes.io~projected/kube-api-access-qx4rg DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/caa5716795758214e8a79342a54eedf4ec7ced420359018bbd53847e75eb49a3/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4d7da57858bd0907a716dbaac42f1783b0ace83acd5f89ab37219848d1b5d0d9/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5cc65189d640b892290d7cf378248b8fbaeab9d260eac05bb6527647b0fed825/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fd74ba94-86d0-4707-94e5-6822c4ab5751/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1eab122e-07da-4307-b7f2-914cbe2b4237/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8860d3615245d0a04f931cc48968445461dce11d162d2a8665fd127bddd7c6ae/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d05e04a8-3ddf-4261-add0-c2324c6c9e21/volumes/kubernetes.io~projected/kube-api-access-jn8lc DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8f7b8781d9e26722564e26832573d120947b1c44750561ef7848fa85b315d8e9/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7ac81c7-e1f0-4549-886d-d7b8b893b614/volumes/kubernetes.io~projected/kube-api-access-5kcdc DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/924ac066-ff98-4320-9eb2-268f0a39ca33/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:128cefb7c44adc8 MacAddress:d2:f1:65:2e:4e:22 Speed:10000 Mtu:8901} {Name:43de827525a68f6 MacAddress:e2:14:8f:f0:da:92 Speed:10000 Mtu:8901} {Name:4d7da57858bd090 MacAddress:3a:43:f9:1e:a7:b0 Speed:10000 Mtu:8901} {Name:69639bd161d7b51 MacAddress:72:ad:82:d7:ef:1f Speed:10000 Mtu:8901} {Name:6b5f03698ad2b8d MacAddress:4a:c8:10:20:17:f2 Speed:10000 Mtu:8901} {Name:74f894480aa0bf7 MacAddress:be:ea:7f:01:fb:97 Speed:10000 Mtu:8901} {Name:837040638b1b42b MacAddress:12:09:90:9d:09:41 Speed:10000 Mtu:8901} {Name:8f7b8781d9e2672 MacAddress:56:3d:8b:2e:ee:74 Speed:10000 Mtu:8901} {Name:9428e795c347bf4 MacAddress:fe:23:e3:75:fc:8e Speed:10000 Mtu:8901} {Name:b333be98afc8504 MacAddress:c2:8d:64:ff:db:db Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f3:3a:c0:9d Speed:0 Mtu:9001} {Name:br-int MacAddress:de:11:c3:b8:a0:68 Speed:0 Mtu:8901} {Name:caa571679575821 MacAddress:72:27:44:b0:4e:ca Speed:10000 Mtu:8901} {Name:d46a7f3e70d6690 MacAddress:fa:81:31:14:fa:5e Speed:10000 Mtu:8901} {Name:e8a68f579b2467a MacAddress:f6:ae:91:84:82:54 Speed:10000 Mtu:8901} {Name:eefca769132f66e MacAddress:ae:cc:12:bd:5e:a2 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f3:3a:c0:9d Speed:0 Mtu:9001} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:a6:d7:56:08:e3:5f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 17:12:22.045187 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.042809 4965 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 17:12:22.045187 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.042854 4965 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 17:12:22.045187 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.043133 4965 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 17:12:22.045187 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.043153 4965 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-222.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 17:12:22.045187 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.043275 4965 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 17:12:22.045187 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.043284 4965 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 17:12:22.045187 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.043294 4965 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:12:22.045187 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.043312 4965 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:12:22.045187 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.043330 4965 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 17:12:22.045187 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.043469 4965 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 17:12:22.045187 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.043580 4965 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 17:12:22.045187 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.043613 4965 kubelet.go:493] "Attempting to sync node with API server" Oct 30 17:12:22.045187 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.043625 4965 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 17:12:22.045187 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.043634 4965 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 17:12:22.045187 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.043643 4965 kubelet.go:405] "Adding apiserver pod source" Oct 30 17:12:22.045187 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.043651 4965 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 17:12:22.046695 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.046674 4965 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 17:12:22.046695 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.046693 4965 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:12:22.047229 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.047210 4965 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 17:12:22.049315 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.049301 4965 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 17:12:22.049315 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.049314 4965 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:12:22.049418 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.049401 4965 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 17:12:22.049587 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.049576 4965 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 17:12:22.049654 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.049604 4965 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 17:12:22.049701 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.049693 4965 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 17:12:22.049753 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.049704 4965 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 17:12:22.049753 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.049711 4965 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 17:12:22.049753 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.049716 4965 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 17:12:22.049753 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.049722 4965 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 17:12:22.049753 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.049728 4965 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 17:12:22.049753 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.049733 4965 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 17:12:22.049753 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.049750 4965 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 17:12:22.049753 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.049757 4965 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 17:12:22.050112 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.049763 4965 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 17:12:22.050112 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.049777 4965 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 17:12:22.050112 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.049787 4965 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 17:12:22.050112 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.049814 4965 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 17:12:22.050112 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.049820 4965 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 17:12:22.053268 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.053255 4965 server.go:1263] "Started kubelet" Oct 30 17:12:22.054423 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.053334 4965 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 17:12:22.054423 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.053698 4965 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 17:12:22.054423 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.053768 4965 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 17:12:22.053806 ip-10-0-139-222 systemd[1]: Started Kubernetes Kubelet. Oct 30 17:12:22.054813 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.054551 4965 server.go:317] "Adding debug handlers to kubelet server" Oct 30 17:12:22.054933 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.054903 4965 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 17:12:22.057668 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.057637 4965 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 17:12:22.059307 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.058269 4965 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 17:12:22.059307 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.058615 4965 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:06:43 +0000 UTC" deadline="2028-04-21 03:10:35.100521677 +0000 UTC" Oct 30 17:12:22.059307 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.058670 4965 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12921h58m13.041855558s" Oct 30 17:12:22.059307 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.058762 4965 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 17:12:22.059307 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.058777 4965 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 17:12:22.059307 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.058886 4965 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 17:12:22.059307 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:22.058989 4965 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-222.ec2.internal\" not found" Oct 30 17:12:22.059825 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.059741 4965 factory.go:55] Registering systemd factory Oct 30 17:12:22.059825 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.059762 4965 factory.go:223] Registration of the systemd container factory successfully Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.060987 4965 factory.go:153] Registering CRI-O factory Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.061006 4965 factory.go:223] Registration of the crio container factory successfully Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.061058 4965 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.061076 4965 factory.go:103] Registering Raw factory Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.061098 4965 manager.go:1199] Started watching for new ooms in manager Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.061457 4965 manager.go:319] Starting recovery of all containers Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.062298 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f442e60f-8ccf-485e-9709-60a909d482bc" volumeName="kubernetes.io/secret/f442e60f-8ccf-485e-9709-60a909d482bc-agent-certs" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.062380 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13444fae-8e71-41ee-8489-55b8606492fb" volumeName="kubernetes.io/projected/13444fae-8e71-41ee-8489-55b8606492fb-kube-api-access-drkzm" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.062424 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd74ba94-86d0-4707-94e5-6822c4ab5751" volumeName="kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-bound-sa-token" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.062440 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="467187ee-7039-4bb4-adbe-38161ee7c336" volumeName="kubernetes.io/empty-dir/467187ee-7039-4bb4-adbe-38161ee7c336-tmp-dir" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.062454 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6084e9b4-6b79-46ab-91cf-8d292d1f4591" volumeName="kubernetes.io/empty-dir/6084e9b4-6b79-46ab-91cf-8d292d1f4591-data-volume" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.062497 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6084e9b4-6b79-46ab-91cf-8d292d1f4591" volumeName="kubernetes.io/projected/6084e9b4-6b79-46ab-91cf-8d292d1f4591-kube-api-access-9zxfh" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.062510 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd74ba94-86d0-4707-94e5-6822c4ab5751" volumeName="kubernetes.io/empty-dir/fd74ba94-86d0-4707-94e5-6822c4ab5751-ca-trust-extracted" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.062542 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16cd937e-4c6d-45d0-a9de-f77dc836d082" volumeName="kubernetes.io/empty-dir/16cd937e-4c6d-45d0-a9de-f77dc836d082-tmp" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.062591 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16cd937e-4c6d-45d0-a9de-f77dc836d082" volumeName="kubernetes.io/projected/16cd937e-4c6d-45d0-a9de-f77dc836d082-kube-api-access-wtlv4" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.062617 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc71466d-eac0-460b-af50-3204f07b3f35" volumeName="kubernetes.io/projected/cc71466d-eac0-460b-af50-3204f07b3f35-kube-api-access-rttdq" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.062645 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d05e04a8-3ddf-4261-add0-c2324c6c9e21" volumeName="kubernetes.io/projected/d05e04a8-3ddf-4261-add0-c2324c6c9e21-kube-api-access-jn8lc" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.062681 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f608efeb-1bda-4375-a428-c6db6341fd0b" volumeName="kubernetes.io/configmap/f608efeb-1bda-4375-a428-c6db6341fd0b-ovnkube-script-lib" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.062701 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2939b3a2-1950-44c9-802c-a53861aa7aa8" volumeName="kubernetes.io/projected/2939b3a2-1950-44c9-802c-a53861aa7aa8-kube-api-access-ffnbr" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.062754 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bdc401b-ae3f-48c0-8110-8cd6096007d8" volumeName="kubernetes.io/projected/4bdc401b-ae3f-48c0-8110-8cd6096007d8-kube-api-access-xkvxq" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.062783 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55f62602-c7b7-4a6c-b244-f6e48775733f" volumeName="kubernetes.io/projected/55f62602-c7b7-4a6c-b244-f6e48775733f-kube-api-access-m9swv" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.062802 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="745f4f0f-74f7-4fff-9384-c8f45f601e33" volumeName="kubernetes.io/projected/745f4f0f-74f7-4fff-9384-c8f45f601e33-kube-api-access-ztbbl" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063141 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="933be7f4-8d62-4735-807f-7aa1439583fa" volumeName="kubernetes.io/configmap/933be7f4-8d62-4735-807f-7aa1439583fa-iptables-alerter-script" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063189 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="933be7f4-8d62-4735-807f-7aa1439583fa" volumeName="kubernetes.io/projected/933be7f4-8d62-4735-807f-7aa1439583fa-kube-api-access-6sh5q" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063212 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="195f952d-ee96-485d-99aa-8b47e6633c97" volumeName="kubernetes.io/projected/195f952d-ee96-485d-99aa-8b47e6633c97-kube-api-access-l25lh" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063232 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16cd937e-4c6d-45d0-a9de-f77dc836d082" volumeName="kubernetes.io/empty-dir/16cd937e-4c6d-45d0-a9de-f77dc836d082-snapshots" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063257 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="924ac066-ff98-4320-9eb2-268f0a39ca33" volumeName="kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-stats-auth" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063277 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce" volumeName="kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-bound-sa-token" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063294 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d05e04a8-3ddf-4261-add0-c2324c6c9e21" volumeName="kubernetes.io/empty-dir/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-tuned" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063313 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f608efeb-1bda-4375-a428-c6db6341fd0b" volumeName="kubernetes.io/configmap/f608efeb-1bda-4375-a428-c6db6341fd0b-ovnkube-config" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063330 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="195f952d-ee96-485d-99aa-8b47e6633c97" volumeName="kubernetes.io/secret/195f952d-ee96-485d-99aa-8b47e6633c97-serving-cert" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063350 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2939b3a2-1950-44c9-802c-a53861aa7aa8" volumeName="kubernetes.io/empty-dir/2939b3a2-1950-44c9-802c-a53861aa7aa8-tmp-dir" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063372 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="745f4f0f-74f7-4fff-9384-c8f45f601e33" volumeName="kubernetes.io/configmap/745f4f0f-74f7-4fff-9384-c8f45f601e33-samples-operator-config" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063386 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce" volumeName="kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-kube-api-access-brgv8" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063419 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7ac81c7-e1f0-4549-886d-d7b8b893b614" volumeName="kubernetes.io/projected/c7ac81c7-e1f0-4549-886d-d7b8b893b614-kube-api-access-5kcdc" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063443 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd6ddeda-aec9-4872-9e43-084577cfcd7d" volumeName="kubernetes.io/configmap/cd6ddeda-aec9-4872-9e43-084577cfcd7d-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063458 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d05e04a8-3ddf-4261-add0-c2324c6c9e21" volumeName="kubernetes.io/empty-dir/d05e04a8-3ddf-4261-add0-c2324c6c9e21-tmp" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063477 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f608efeb-1bda-4375-a428-c6db6341fd0b" volumeName="kubernetes.io/secret/f608efeb-1bda-4375-a428-c6db6341fd0b-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063549 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27ac623c-2f57-44ec-a1b1-3ad87909452d" volumeName="kubernetes.io/secret/27ac623c-2f57-44ec-a1b1-3ad87909452d-serving-cert" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063577 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13444fae-8e71-41ee-8489-55b8606492fb" volumeName="kubernetes.io/configmap/13444fae-8e71-41ee-8489-55b8606492fb-trusted-ca-bundle" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063606 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55f62602-c7b7-4a6c-b244-f6e48775733f" volumeName="kubernetes.io/configmap/55f62602-c7b7-4a6c-b244-f6e48775733f-cni-binary-copy" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063630 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89eff971-c01b-4b06-9c16-4332956558a3" volumeName="kubernetes.io/projected/89eff971-c01b-4b06-9c16-4332956558a3-kube-api-access-ddss9" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063650 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce" volumeName="kubernetes.io/empty-dir/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-ca-trust-extracted" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063676 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd6ddeda-aec9-4872-9e43-084577cfcd7d" volumeName="kubernetes.io/projected/cd6ddeda-aec9-4872-9e43-084577cfcd7d-kube-api-access-r67qt" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063694 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd74ba94-86d0-4707-94e5-6822c4ab5751" volumeName="kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-kube-api-access-qx4rg" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063722 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd74ba94-86d0-4707-94e5-6822c4ab5751" volumeName="kubernetes.io/secret/fd74ba94-86d0-4707-94e5-6822c4ab5751-installation-pull-secrets" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063747 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16cd937e-4c6d-45d0-a9de-f77dc836d082" volumeName="kubernetes.io/configmap/16cd937e-4c6d-45d0-a9de-f77dc836d082-trusted-ca-bundle" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063764 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="467187ee-7039-4bb4-adbe-38161ee7c336" volumeName="kubernetes.io/projected/467187ee-7039-4bb4-adbe-38161ee7c336-kube-api-access-p7n64" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063795 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55f62602-c7b7-4a6c-b244-f6e48775733f" volumeName="kubernetes.io/configmap/55f62602-c7b7-4a6c-b244-f6e48775733f-multus-daemon-config" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063811 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce" volumeName="kubernetes.io/configmap/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-certificates" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063845 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce" volumeName="kubernetes.io/secret/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-installation-pull-secrets" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063877 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f442e60f-8ccf-485e-9709-60a909d482bc" volumeName="kubernetes.io/configmap/f442e60f-8ccf-485e-9709-60a909d482bc-konnectivity-ca" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063896 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2939b3a2-1950-44c9-802c-a53861aa7aa8" volumeName="kubernetes.io/configmap/2939b3a2-1950-44c9-802c-a53861aa7aa8-config-volume" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063923 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89eff971-c01b-4b06-9c16-4332956558a3" volumeName="kubernetes.io/secret/89eff971-c01b-4b06-9c16-4332956558a3-cert" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063939 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2d3293d-1890-4fa2-b790-9f84622c9c8b" volumeName="kubernetes.io/configmap/c2d3293d-1890-4fa2-b790-9f84622c9c8b-nginx-conf" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063966 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e30e6b59-b8e2-4125-ae7a-41edd1ce7389" volumeName="kubernetes.io/projected/e30e6b59-b8e2-4125-ae7a-41edd1ce7389-kube-api-access-5dz8c" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063982 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5f38da3-fe1f-4f36-8ed3-12cb01bf128b" volumeName="kubernetes.io/projected/e5f38da3-fe1f-4f36-8ed3-12cb01bf128b-kube-api-access-5zz48" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.063999 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd74ba94-86d0-4707-94e5-6822c4ab5751" volumeName="kubernetes.io/secret/fd74ba94-86d0-4707-94e5-6822c4ab5751-image-registry-private-configuration" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064031 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03a0e7f9-9e86-4aa9-9864-205263f6a486" volumeName="kubernetes.io/projected/03a0e7f9-9e86-4aa9-9864-205263f6a486-kube-api-access-txhrv" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064048 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27ac623c-2f57-44ec-a1b1-3ad87909452d" volumeName="kubernetes.io/configmap/27ac623c-2f57-44ec-a1b1-3ad87909452d-config" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064078 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bdc401b-ae3f-48c0-8110-8cd6096007d8" volumeName="kubernetes.io/configmap/4bdc401b-ae3f-48c0-8110-8cd6096007d8-signing-cabundle" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064097 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6178534d-9f75-49c7-a977-de65737cbcea" volumeName="kubernetes.io/projected/6178534d-9f75-49c7-a977-de65737cbcea-kube-api-access-nqrt8" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064145 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d39ddd3a-781f-4084-b4e2-624b130c1025" volumeName="kubernetes.io/projected/d39ddd3a-781f-4084-b4e2-624b130c1025-kube-api-access-z4njs" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064170 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f608efeb-1bda-4375-a428-c6db6341fd0b" volumeName="kubernetes.io/projected/f608efeb-1bda-4375-a428-c6db6341fd0b-kube-api-access-d49vx" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064195 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2906e95e-3baa-4c79-ad9c-b2592c1d414f" volumeName="kubernetes.io/secret/2906e95e-3baa-4c79-ad9c-b2592c1d414f-original-pull-secret" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064218 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13444fae-8e71-41ee-8489-55b8606492fb" volumeName="kubernetes.io/configmap/13444fae-8e71-41ee-8489-55b8606492fb-service-ca-bundle" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064237 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1eab122e-07da-4307-b7f2-914cbe2b4237" volumeName="kubernetes.io/configmap/1eab122e-07da-4307-b7f2-914cbe2b4237-config" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064253 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc71466d-eac0-460b-af50-3204f07b3f35" volumeName="kubernetes.io/configmap/cc71466d-eac0-460b-af50-3204f07b3f35-telemetry-config" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064278 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd6ddeda-aec9-4872-9e43-084577cfcd7d" volumeName="kubernetes.io/configmap/cd6ddeda-aec9-4872-9e43-084577cfcd7d-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064294 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2664a881-44d7-470a-93d0-df1b3eaca0bc" volumeName="kubernetes.io/configmap/2664a881-44d7-470a-93d0-df1b3eaca0bc-serviceca" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064321 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2989c0bb-0ed6-41b2-ad3e-bb6e85bd60b9" volumeName="kubernetes.io/projected/2989c0bb-0ed6-41b2-ad3e-bb6e85bd60b9-kube-api-access-csqbr" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064338 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bdc401b-ae3f-48c0-8110-8cd6096007d8" volumeName="kubernetes.io/secret/4bdc401b-ae3f-48c0-8110-8cd6096007d8-signing-key" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064362 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd74ba94-86d0-4707-94e5-6822c4ab5751" volumeName="kubernetes.io/configmap/fd74ba94-86d0-4707-94e5-6822c4ab5751-trusted-ca" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064386 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16cd937e-4c6d-45d0-a9de-f77dc836d082" volumeName="kubernetes.io/configmap/16cd937e-4c6d-45d0-a9de-f77dc836d082-service-ca-bundle" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064403 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1eab122e-07da-4307-b7f2-914cbe2b4237" volumeName="kubernetes.io/configmap/1eab122e-07da-4307-b7f2-914cbe2b4237-trusted-ca" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064425 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1eab122e-07da-4307-b7f2-914cbe2b4237" volumeName="kubernetes.io/secret/1eab122e-07da-4307-b7f2-914cbe2b4237-serving-cert" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064444 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd74ba94-86d0-4707-94e5-6822c4ab5751" volumeName="kubernetes.io/configmap/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-certificates" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064460 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2664a881-44d7-470a-93d0-df1b3eaca0bc" volumeName="kubernetes.io/projected/2664a881-44d7-470a-93d0-df1b3eaca0bc-kube-api-access-p5mmm" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064483 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce" volumeName="kubernetes.io/configmap/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-trusted-ca" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064502 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7b7edd7-11cc-4fac-92ae-a08140056adb" volumeName="kubernetes.io/projected/c7b7edd7-11cc-4fac-92ae-a08140056adb-kube-api-access-xs2g4" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064532 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f608efeb-1bda-4375-a428-c6db6341fd0b" volumeName="kubernetes.io/configmap/f608efeb-1bda-4375-a428-c6db6341fd0b-env-overrides" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064550 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="195f952d-ee96-485d-99aa-8b47e6633c97" volumeName="kubernetes.io/configmap/195f952d-ee96-485d-99aa-8b47e6633c97-config" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064570 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13444fae-8e71-41ee-8489-55b8606492fb" volumeName="kubernetes.io/empty-dir/13444fae-8e71-41ee-8489-55b8606492fb-archives-path" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064593 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1eab122e-07da-4307-b7f2-914cbe2b4237" volumeName="kubernetes.io/projected/1eab122e-07da-4307-b7f2-914cbe2b4237-kube-api-access-f9kmm" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064610 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="924ac066-ff98-4320-9eb2-268f0a39ca33" volumeName="kubernetes.io/projected/924ac066-ff98-4320-9eb2-268f0a39ca33-kube-api-access-mrqrz" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064635 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce" volumeName="kubernetes.io/secret/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-image-registry-private-configuration" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064654 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd6ddeda-aec9-4872-9e43-084577cfcd7d" volumeName="kubernetes.io/configmap/cd6ddeda-aec9-4872-9e43-084577cfcd7d-cni-binary-copy" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064672 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16cd937e-4c6d-45d0-a9de-f77dc836d082" volumeName="kubernetes.io/secret/16cd937e-4c6d-45d0-a9de-f77dc836d082-serving-cert" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064706 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27ac623c-2f57-44ec-a1b1-3ad87909452d" volumeName="kubernetes.io/projected/27ac623c-2f57-44ec-a1b1-3ad87909452d-kube-api-access-g4qm2" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064726 4965 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="924ac066-ff98-4320-9eb2-268f0a39ca33" volumeName="kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-default-certificate" seLinuxMountContext="" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064751 4965 reconstruct.go:97] "Volume reconstruction finished" Oct 30 17:12:22.065848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.064761 4965 reconciler.go:29] "Reconciler: start to sync state" Oct 30 17:12:22.081613 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.081589 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 17:12:22.081857 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.081840 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 17:12:22.088580 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.088536 4965 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 17:12:22.090158 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.090141 4965 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 17:12:22.090257 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.090166 4965 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 17:12:22.090257 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.090192 4965 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 17:12:22.090257 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:22.090225 4965 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 17:12:22.102511 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.102486 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 17:12:22.102605 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.102523 4965 generic.go:356] "Generic (PLEG): container finished" podID="1eab122e-07da-4307-b7f2-914cbe2b4237" containerID="c562428fe21e43cf22e26ebeba52550c57e2741dc839171b41e59c2d589d4922" exitCode=255 Oct 30 17:12:22.106565 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.106544 4965 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 17:12:22.107465 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.107399 4965 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:12:22.107819 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.107802 4965 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:12:22.120187 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.120164 4965 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:12:22.120546 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.120523 4965 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:12:22.131299 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.131227 4965 generic.go:356] "Generic (PLEG): container finished" podID="cd6ddeda-aec9-4872-9e43-084577cfcd7d" containerID="3c3c9ee9b09b99e5d5170d9644f47e4a5bf6af86e22f4fb3ae61b125c57c0d2a" exitCode=0 Oct 30 17:12:22.131299 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.131254 4965 generic.go:356] "Generic (PLEG): container finished" podID="cd6ddeda-aec9-4872-9e43-084577cfcd7d" containerID="44e59dce5e2ddb6caf2141b8bf3e9dc8c70c93ca53ed78e9abbff5983e8acf5c" exitCode=0 Oct 30 17:12:22.131299 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.131261 4965 generic.go:356] "Generic (PLEG): container finished" podID="cd6ddeda-aec9-4872-9e43-084577cfcd7d" containerID="2301b4325aa5093c02513414b0c9db48ebcf49537ce78679d820c41cd4d79c92" exitCode=0 Oct 30 17:12:22.131299 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.131267 4965 generic.go:356] "Generic (PLEG): container finished" podID="cd6ddeda-aec9-4872-9e43-084577cfcd7d" containerID="bd8c48abaf647b0718e65353956a31c298042887a627746b3a40f74dbc639590" exitCode=0 Oct 30 17:12:22.131299 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.131273 4965 generic.go:356] "Generic (PLEG): container finished" podID="cd6ddeda-aec9-4872-9e43-084577cfcd7d" containerID="8fc8fb8996d9593ecdc982cbe18ca846f7c9c44f0bf544fef4e98a0701ffdfe2" exitCode=0 Oct 30 17:12:22.131299 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.131280 4965 generic.go:356] "Generic (PLEG): container finished" podID="cd6ddeda-aec9-4872-9e43-084577cfcd7d" containerID="1b53ed7e67d84f74e2642c1bb64495972105deaeb7c221a30edded531a51c575" exitCode=0 Oct 30 17:12:22.143960 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.143932 4965 generic.go:356] "Generic (PLEG): container finished" podID="d961f94773b0889650472ea1a7f54825" containerID="1f14215c577069a9da506cd135905f95b124159c42614e72daf64985cf8c86c2" exitCode=0 Oct 30 17:12:22.158929 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.158912 4965 manager.go:324] Recovery completed Oct 30 17:12:22.179092 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.179068 4965 cpu_manager.go:226] "Starting" policy="none" Oct 30 17:12:22.179092 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.179089 4965 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 17:12:22.179241 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.179110 4965 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 17:12:22.179287 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.179276 4965 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 17:12:22.179336 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.179290 4965 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 17:12:22.179336 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.179313 4965 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 17:12:22.179336 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.179322 4965 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 17:12:22.179336 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.179329 4965 policy_none.go:50] "Start" Oct 30 17:12:22.179502 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.179348 4965 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 17:12:22.179502 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.179360 4965 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 17:12:22.179589 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.179577 4965 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 17:12:22.179635 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.179587 4965 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 17:12:22.179635 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.179596 4965 policy_none.go:44] "Start" Oct 30 17:12:22.184554 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.184524 4965 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 17:12:22.187255 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.187240 4965 manager.go:341] "Starting Device Plugin manager" Oct 30 17:12:22.187934 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:22.187264 4965 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 17:12:22.187934 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.187935 4965 server.go:87] "Starting device plugin registration server" Oct 30 17:12:22.188263 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.188247 4965 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 17:12:22.188339 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.188262 4965 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 17:12:22.188394 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.188367 4965 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 17:12:22.188538 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.188486 4965 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 17:12:22.188620 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.188541 4965 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 17:12:22.188620 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.188549 4965 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 17:12:22.188720 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.188597 4965 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T17:12:22.188530746Z","UUID":"2a8d7b40-da7c-4962-96f5-821974701b95","Handler":null,"Name":"","Endpoint":""} Oct 30 17:12:22.189711 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.189696 4965 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 17:12:22.189818 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.189727 4965 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 17:12:22.190153 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:22.190133 4965 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 17:12:22.190295 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.190266 4965 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-222.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal"] Oct 30 17:12:22.191084 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.191067 4965 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-222.ec2.internal" Oct 30 17:12:22.191161 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.191106 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-222.ec2.internal" event={"ID":"67161c7401d4577fb0911752be0668c2","Type":"ContainerStarted","Data":"fda2c96be0659a82de123de3859cddbf59249b57f4b433f9f21900d66be20c97"} Oct 30 17:12:22.191161 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.191151 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-222.ec2.internal" event={"ID":"67161c7401d4577fb0911752be0668c2","Type":"ContainerStarted","Data":"b1f4230704e571ae81b3c38071fcfbdcdf9ff5933438e5bdbb3e99520e76bcee"} Oct 30 17:12:22.191315 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.191298 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal" event={"ID":"d961f94773b0889650472ea1a7f54825","Type":"ContainerStarted","Data":"61598f925a2bb36cc40a56cb98ec1a95972e30279ea51b60e459a50b5869b3be"} Oct 30 17:12:22.191375 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.191325 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal" event={"ID":"d961f94773b0889650472ea1a7f54825","Type":"ContainerDied","Data":"1f14215c577069a9da506cd135905f95b124159c42614e72daf64985cf8c86c2"} Oct 30 17:12:22.191375 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.191345 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal" event={"ID":"d961f94773b0889650472ea1a7f54825","Type":"ContainerStarted","Data":"8860d3615245d0a04f931cc48968445461dce11d162d2a8665fd127bddd7c6ae"} Oct 30 17:12:22.191615 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.191579 4965 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal" Oct 30 17:12:22.214281 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.214252 4965 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:12:22.214393 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:22.214297 4965 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-222.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-222.ec2.internal" Oct 30 17:12:22.215083 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.215057 4965 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:12:22.215180 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:22.215126 4965 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal" Oct 30 17:12:22.289139 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.289111 4965 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:22.290736 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.290719 4965 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-222.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:22.290823 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.290747 4965 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-222.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:22.290823 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.290756 4965 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-222.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:22.290823 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.290804 4965 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-222.ec2.internal" Oct 30 17:12:22.307156 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.307134 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d961f94773b0889650472ea1a7f54825-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal\" (UID: \"d961f94773b0889650472ea1a7f54825\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal" Oct 30 17:12:22.307215 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.307168 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d961f94773b0889650472ea1a7f54825-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal\" (UID: \"d961f94773b0889650472ea1a7f54825\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal" Oct 30 17:12:22.307260 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.307237 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/67161c7401d4577fb0911752be0668c2-config\") pod \"kube-apiserver-proxy-ip-10-0-139-222.ec2.internal\" (UID: \"67161c7401d4577fb0911752be0668c2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-222.ec2.internal" Oct 30 17:12:22.326214 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.326179 4965 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-222.ec2.internal" Oct 30 17:12:22.326297 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.326266 4965 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-222.ec2.internal" Oct 30 17:12:22.407831 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.407747 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/67161c7401d4577fb0911752be0668c2-config\") pod \"kube-apiserver-proxy-ip-10-0-139-222.ec2.internal\" (UID: \"67161c7401d4577fb0911752be0668c2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-222.ec2.internal" Oct 30 17:12:22.407831 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.407786 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d961f94773b0889650472ea1a7f54825-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal\" (UID: \"d961f94773b0889650472ea1a7f54825\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal" Oct 30 17:12:22.407831 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.407805 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d961f94773b0889650472ea1a7f54825-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal\" (UID: \"d961f94773b0889650472ea1a7f54825\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal" Oct 30 17:12:22.408065 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.407849 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d961f94773b0889650472ea1a7f54825-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal\" (UID: \"d961f94773b0889650472ea1a7f54825\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal" Oct 30 17:12:22.408065 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.407855 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d961f94773b0889650472ea1a7f54825-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal\" (UID: \"d961f94773b0889650472ea1a7f54825\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal" Oct 30 17:12:22.408065 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:22.407854 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/67161c7401d4577fb0911752be0668c2-config\") pod \"kube-apiserver-proxy-ip-10-0-139-222.ec2.internal\" (UID: \"67161c7401d4577fb0911752be0668c2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-222.ec2.internal" Oct 30 17:12:23.044409 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.044375 4965 apiserver.go:52] "Watching apiserver" Oct 30 17:12:23.085400 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.085370 4965 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 17:12:23.086061 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.086034 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-w54kt","openshift-console-operator/console-operator-7cd56b8dbc-mf9rd","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4x66d","openshift-multus/multus-db6jk","openshift-multus/multus-additional-cni-plugins-7gvrp","openshift-image-registry/image-registry-77579944f4-hctng","openshift-service-ca/service-ca-5fffc97bdf-27cwx","openshift-image-registry/node-ca-stqrx","kube-system/konnectivity-agent-jnfks","openshift-ingress-canary/ingress-canary-rdjn5","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jccch","openshift-network-diagnostics/network-check-source-6f78d59f99-z6b8h","openshift-dns/dns-default-22vlj","kube-system/global-pull-secret-syncer-5mtjr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-222.ec2.internal","openshift-multus/network-metrics-daemon-87jjv","openshift-insights/periodic-gathering-2ftd7-xc6xx","openshift-cluster-node-tuning-operator/tuned-pvpvp","openshift-monitoring/cluster-monitoring-operator-884576c57-6jbgg","openshift-network-console/networking-console-plugin-86c478bb88-kvpw6","openshift-image-registry/image-registry-585b9dc6c5-dblkr","openshift-dns/node-resolver-zgb2b","kube-system/kas-connection-checker-685895c7c9-hwfbh","kube-system/kas-connection-checker-685895c7c9-w5rnm","openshift-network-diagnostics/network-check-target-t2pqm","openshift-ingress/router-default-ddb54c4f7-54tw2","openshift-insights/insights-runtime-extractor-tbqhc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq","openshift-network-operator/iptables-alerter-pj5s7","openshift-service-ca-operator/service-ca-operator-544d749c8c-2tk8w","kube-system/kube-apiserver-proxy-ip-10-0-139-222.ec2.internal","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t9hdn","openshift-insights/insights-operator-7c4986d449-4w8fl","kube-system/kas-connection-checker-685895c7c9-9jx9m"] Oct 30 17:12:23.088418 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.088392 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jbgg" Oct 30 17:12:23.089079 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.089062 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:23.089501 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.089482 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t9hdn" Oct 30 17:12:23.090256 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.090242 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:23.091242 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.091223 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-kvpw6" Oct 30 17:12:23.098429 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.098411 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:12:23.099953 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.099936 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:23.100908 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.100894 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-22vlj" Oct 30 17:12:23.101414 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.101399 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:23.103939 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.103923 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 17:12:23.103939 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.103936 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 17:12:23.107996 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.107975 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 17:12:23.107996 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.107981 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:12:23.108121 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.108032 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 17:12:23.108121 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.108045 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 17:12:23.108121 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.108033 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:12:23.108121 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.108074 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 17:12:23.108121 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.108064 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 17:12:23.108360 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.108129 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 17:12:23.108482 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.108464 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 17:12:23.108657 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.108641 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-5pzbw\"" Oct 30 17:12:23.108832 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.108817 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 17:12:23.108916 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.108896 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hr7mp\"" Oct 30 17:12:23.108958 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.108942 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-xfcrk\"" Oct 30 17:12:23.109087 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.109071 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 17:12:23.109148 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.109106 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 17:12:23.109307 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.109294 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 17:12:23.114544 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.114524 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 17:12:23.114687 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.114671 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 17:12:23.114687 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.114685 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 17:12:23.114800 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.114787 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:12:23.115059 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.115045 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 17:12:23.115104 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.115069 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 17:12:23.115287 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.115267 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 17:12:23.115287 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.115283 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:12:23.115422 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.115290 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 17:12:23.115422 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.115303 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 17:12:23.115422 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.115335 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:12:23.115422 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.115376 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 17:12:23.120996 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.120955 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 17:12:23.120996 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.120966 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 17:12:23.121143 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.121000 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 17:12:23.121143 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.121031 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 17:12:23.121143 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.121044 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 17:12:23.121143 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.121045 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 17:12:23.121143 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.121079 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 17:12:23.121143 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.120965 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 17:12:23.121143 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.121110 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 17:12:23.121143 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.121126 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 17:12:23.121515 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.121151 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 17:12:23.121515 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.121156 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 17:12:23.121515 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.121164 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:12:23.121515 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.121221 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 17:12:23.121515 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.121280 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:12:23.121515 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.121444 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 17:12:23.121515 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.121458 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 17:12:23.121958 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.121942 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 17:12:23.122066 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.121967 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 17:12:23.122066 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.121988 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 17:12:23.122066 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.122034 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 17:12:23.122219 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.122098 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 17:12:23.122219 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.122171 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 17:12:23.122629 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.122605 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 17:12:23.122702 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.122640 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 17:12:23.122702 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.122675 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 17:12:23.123152 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.123131 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 17:12:23.123304 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.123140 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 17:12:23.123469 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.123450 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 17:12:23.123618 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.123602 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 17:12:23.123888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.123854 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 17:12:23.123968 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.123901 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 17:12:23.123968 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.123964 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 17:12:23.124076 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.124050 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 17:12:23.124519 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.124501 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 17:12:23.124588 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.124564 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 17:12:23.124648 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.124628 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 17:12:23.124691 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.124650 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 17:12:23.124691 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.124676 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 17:12:23.125046 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.125028 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-8nbr8\"" Oct 30 17:12:23.125218 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.125201 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 17:12:23.125444 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.125428 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 17:12:23.125555 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.125537 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 17:12:23.125624 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.125608 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 17:12:23.125723 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.125710 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 17:12:23.125885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.125847 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 17:12:23.125991 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.125977 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-786df\"" Oct 30 17:12:23.126144 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.126125 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mkzvv\"" Oct 30 17:12:23.126212 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.126136 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qn7sb\"" Oct 30 17:12:23.126212 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.126204 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-qjw5w\"" Oct 30 17:12:23.126313 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.126247 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2wbm5\"" Oct 30 17:12:23.126313 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.126256 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8vxcg\"" Oct 30 17:12:23.126313 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.126293 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-599zc\"" Oct 30 17:12:23.126313 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.125441 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-r5hq7\"" Oct 30 17:12:23.126483 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.125722 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 17:12:23.126483 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.126353 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-bvkn4\"" Oct 30 17:12:23.126483 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.126447 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 17:12:23.126598 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.126513 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 17:12:23.126598 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.126578 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 17:12:23.126598 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.126583 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n8szd\"" Oct 30 17:12:23.126741 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.126728 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 17:12:23.126902 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.126877 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 17:12:23.127309 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.127295 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rz5ks\"" Oct 30 17:12:23.127378 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.127312 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 17:12:23.127695 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.127676 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-7qw5s\"" Oct 30 17:12:23.127773 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.127712 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9lrx5\"" Oct 30 17:12:23.127773 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.127724 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6mz24\"" Oct 30 17:12:23.127773 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.127733 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-g55vh\"" Oct 30 17:12:23.127773 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.127735 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-49f6v\"" Oct 30 17:12:23.127773 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.127765 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ctbv7\"" Oct 30 17:12:23.127997 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.127795 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zkp55\"" Oct 30 17:12:23.127997 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.127820 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-n92rc\"" Oct 30 17:12:23.127997 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.127836 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 17:12:23.127997 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.127843 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-pnbwd\"" Oct 30 17:12:23.127997 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.127896 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-j8vzc\"" Oct 30 17:12:23.127997 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.127958 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 17:12:23.127997 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.127966 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jj2qr\"" Oct 30 17:12:23.128212 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.128009 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 17:12:23.128212 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.128082 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wklnr\"" Oct 30 17:12:23.128212 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.128138 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 17:12:23.128212 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.128193 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 17:12:23.146959 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.146939 4965 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-222.ec2.internal" Oct 30 17:12:23.161584 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.161567 4965 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 17:12:23.167647 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.167632 4965 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:12:23.167728 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:23.167672 4965 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-222.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-222.ec2.internal" Oct 30 17:12:23.212905 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.212882 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13444fae-8e71-41ee-8489-55b8606492fb-service-ca-bundle\") pod \"periodic-gathering-2ftd7-xc6xx\" (UID: \"13444fae-8e71-41ee-8489-55b8606492fb\") " pod="openshift-insights/periodic-gathering-2ftd7-xc6xx" Oct 30 17:12:23.213007 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.212909 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkvxq\" (UniqueName: \"kubernetes.io/projected/4bdc401b-ae3f-48c0-8110-8cd6096007d8-kube-api-access-xkvxq\") pod \"service-ca-5fffc97bdf-27cwx\" (UID: \"4bdc401b-ae3f-48c0-8110-8cd6096007d8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-27cwx" Oct 30 17:12:23.213007 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.212928 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7n64\" (UniqueName: \"kubernetes.io/projected/467187ee-7039-4bb4-adbe-38161ee7c336-kube-api-access-p7n64\") pod \"node-resolver-zgb2b\" (UID: \"467187ee-7039-4bb4-adbe-38161ee7c336\") " pod="openshift-dns/node-resolver-zgb2b" Oct 30 17:12:23.213007 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.212973 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f608efeb-1bda-4375-a428-c6db6341fd0b-ovnkube-script-lib\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.213007 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.213003 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1eab122e-07da-4307-b7f2-914cbe2b4237-trusted-ca\") pod \"console-operator-7cd56b8dbc-mf9rd\" (UID: \"1eab122e-07da-4307-b7f2-914cbe2b4237\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" Oct 30 17:12:23.213204 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.213025 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc71466d-eac0-460b-af50-3204f07b3f35-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6jbgg\" (UID: \"cc71466d-eac0-460b-af50-3204f07b3f35\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jbgg" Oct 30 17:12:23.213250 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.213207 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6084e9b4-6b79-46ab-91cf-8d292d1f4591-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:23.213293 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.213246 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13444fae-8e71-41ee-8489-55b8606492fb-service-ca-bundle\") pod \"periodic-gathering-2ftd7-xc6xx\" (UID: \"13444fae-8e71-41ee-8489-55b8606492fb\") " pod="openshift-insights/periodic-gathering-2ftd7-xc6xx" Oct 30 17:12:23.213293 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.213261 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-systemd-units\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.213388 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.213306 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/195f952d-ee96-485d-99aa-8b47e6633c97-config\") pod \"service-ca-operator-544d749c8c-2tk8w\" (UID: \"195f952d-ee96-485d-99aa-8b47e6633c97\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2tk8w" Oct 30 17:12:23.213388 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.213343 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zz48\" (UniqueName: \"kubernetes.io/projected/e5f38da3-fe1f-4f36-8ed3-12cb01bf128b-kube-api-access-5zz48\") pod \"network-check-source-6f78d59f99-z6b8h\" (UID: \"e5f38da3-fe1f-4f36-8ed3-12cb01bf128b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z6b8h" Oct 30 17:12:23.213388 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.213360 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1eab122e-07da-4307-b7f2-914cbe2b4237-trusted-ca\") pod \"console-operator-7cd56b8dbc-mf9rd\" (UID: \"1eab122e-07da-4307-b7f2-914cbe2b4237\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" Oct 30 17:12:23.213528 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.213423 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f608efeb-1bda-4375-a428-c6db6341fd0b-ovnkube-script-lib\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.213528 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.213449 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn8lc\" (UniqueName: \"kubernetes.io/projected/d05e04a8-3ddf-4261-add0-c2324c6c9e21-kube-api-access-jn8lc\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.213528 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.213456 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/195f952d-ee96-485d-99aa-8b47e6633c97-config\") pod \"service-ca-operator-544d749c8c-2tk8w\" (UID: \"195f952d-ee96-485d-99aa-8b47e6633c97\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2tk8w" Oct 30 17:12:23.213656 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.213551 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4njs\" (UniqueName: \"kubernetes.io/projected/d39ddd3a-781f-4084-b4e2-624b130c1025-kube-api-access-z4njs\") pod \"network-check-target-t2pqm\" (UID: \"d39ddd3a-781f-4084-b4e2-624b130c1025\") " pod="openshift-network-diagnostics/network-check-target-t2pqm" Oct 30 17:12:23.213656 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.213576 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-modprobe-d\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.213656 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.213614 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-tuned\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.213656 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.213641 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/924ac066-ff98-4320-9eb2-268f0a39ca33-service-ca-bundle\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:23.213825 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.213666 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrqrz\" (UniqueName: \"kubernetes.io/projected/924ac066-ff98-4320-9eb2-268f0a39ca33-kube-api-access-mrqrz\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:23.213825 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.213702 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c2d3293d-1890-4fa2-b790-9f84622c9c8b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-kvpw6\" (UID: \"c2d3293d-1890-4fa2-b790-9f84622c9c8b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kvpw6" Oct 30 17:12:23.213825 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.213790 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-etc-selinux\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:12:23.213825 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.213782 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-tuned\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.214089 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.213827 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-node-log\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.214089 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.213842 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c2d3293d-1890-4fa2-b790-9f84622c9c8b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-kvpw6\" (UID: \"c2d3293d-1890-4fa2-b790-9f84622c9c8b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kvpw6" Oct 30 17:12:23.214089 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.213875 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx4rg\" (UniqueName: \"kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-kube-api-access-qx4rg\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:23.214089 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.213914 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4qm2\" (UniqueName: \"kubernetes.io/projected/27ac623c-2f57-44ec-a1b1-3ad87909452d-kube-api-access-g4qm2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jccch\" (UID: \"27ac623c-2f57-44ec-a1b1-3ad87909452d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jccch" Oct 30 17:12:23.214089 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.213966 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd6ddeda-aec9-4872-9e43-084577cfcd7d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:12:23.214089 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.213993 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-hostroot\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.214089 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.214016 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-etc-kubernetes\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.214089 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.214044 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztbbl\" (UniqueName: \"kubernetes.io/projected/745f4f0f-74f7-4fff-9384-c8f45f601e33-kube-api-access-ztbbl\") pod \"cluster-samples-operator-6ddb6c8b94-t9hdn\" (UID: \"745f4f0f-74f7-4fff-9384-c8f45f601e33\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t9hdn" Oct 30 17:12:23.214089 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.214075 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-installation-pull-secrets\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:23.214443 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.214142 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-certificates\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:23.214443 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.214170 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqrt8\" (UniqueName: \"kubernetes.io/projected/6178534d-9f75-49c7-a977-de65737cbcea-kube-api-access-nqrt8\") pod \"network-metrics-daemon-87jjv\" (UID: \"6178534d-9f75-49c7-a977-de65737cbcea\") " pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:12:23.214443 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.214203 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zxfh\" (UniqueName: \"kubernetes.io/projected/6084e9b4-6b79-46ab-91cf-8d292d1f4591-kube-api-access-9zxfh\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:23.214443 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.214281 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-installation-pull-secrets\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:23.214443 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.214345 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd6ddeda-aec9-4872-9e43-084577cfcd7d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:12:23.214443 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.214365 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd6ddeda-aec9-4872-9e43-084577cfcd7d-system-cni-dir\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:12:23.214443 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.214405 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/745f4f0f-74f7-4fff-9384-c8f45f601e33-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-t9hdn\" (UID: \"745f4f0f-74f7-4fff-9384-c8f45f601e33\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t9hdn" Oct 30 17:12:23.214443 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.214427 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd74ba94-86d0-4707-94e5-6822c4ab5751-trusted-ca\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:23.214823 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.214545 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f442e60f-8ccf-485e-9709-60a909d482bc-agent-certs\") pod \"konnectivity-agent-jnfks\" (UID: \"f442e60f-8ccf-485e-9709-60a909d482bc\") " pod="kube-system/konnectivity-agent-jnfks" Oct 30 17:12:23.214823 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.214576 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-host\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.214823 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.214593 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/745f4f0f-74f7-4fff-9384-c8f45f601e33-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-t9hdn\" (UID: \"745f4f0f-74f7-4fff-9384-c8f45f601e33\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t9hdn" Oct 30 17:12:23.214823 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.214608 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd6ddeda-aec9-4872-9e43-084577cfcd7d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:12:23.214823 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.214643 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55f62602-c7b7-4a6c-b244-f6e48775733f-cni-binary-copy\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.214823 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.214758 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd74ba94-86d0-4707-94e5-6822c4ab5751-trusted-ca\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:23.214823 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.214801 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-run-k8s-cni-cncf-io\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.214823 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.214814 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f442e60f-8ccf-485e-9709-60a909d482bc-agent-certs\") pod \"konnectivity-agent-jnfks\" (UID: \"f442e60f-8ccf-485e-9709-60a909d482bc\") " pod="kube-system/konnectivity-agent-jnfks" Oct 30 17:12:23.215231 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.214834 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55f62602-c7b7-4a6c-b244-f6e48775733f-cni-binary-copy\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.215231 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.214846 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-bound-sa-token\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:23.215231 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.214856 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd6ddeda-aec9-4872-9e43-084577cfcd7d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:12:23.215231 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.214898 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dz8c\" (UniqueName: \"kubernetes.io/projected/e30e6b59-b8e2-4125-ae7a-41edd1ce7389-kube-api-access-5dz8c\") pod \"kas-connection-checker-685895c7c9-hwfbh\" (UID: \"e30e6b59-b8e2-4125-ae7a-41edd1ce7389\") " pod="kube-system/kas-connection-checker-685895c7c9-hwfbh" Oct 30 17:12:23.215231 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.214919 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drkzm\" (UniqueName: \"kubernetes.io/projected/13444fae-8e71-41ee-8489-55b8606492fb-kube-api-access-drkzm\") pod \"periodic-gathering-2ftd7-xc6xx\" (UID: \"13444fae-8e71-41ee-8489-55b8606492fb\") " pod="openshift-insights/periodic-gathering-2ftd7-xc6xx" Oct 30 17:12:23.215231 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.214936 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-sys\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.215231 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.214954 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-device-dir\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:12:23.215231 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.214977 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/467187ee-7039-4bb4-adbe-38161ee7c336-tmp-dir\") pod \"node-resolver-zgb2b\" (UID: \"467187ee-7039-4bb4-adbe-38161ee7c336\") " pod="openshift-dns/node-resolver-zgb2b" Oct 30 17:12:23.215231 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215071 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-certificates\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:23.215231 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215076 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-cni-netd\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.215231 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215128 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/467187ee-7039-4bb4-adbe-38161ee7c336-tmp-dir\") pod \"node-resolver-zgb2b\" (UID: \"467187ee-7039-4bb4-adbe-38161ee7c336\") " pod="openshift-dns/node-resolver-zgb2b" Oct 30 17:12:23.215231 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215147 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/467187ee-7039-4bb4-adbe-38161ee7c336-hosts-file\") pod \"node-resolver-zgb2b\" (UID: \"467187ee-7039-4bb4-adbe-38161ee7c336\") " pod="openshift-dns/node-resolver-zgb2b" Oct 30 17:12:23.215231 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215170 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/195f952d-ee96-485d-99aa-8b47e6633c97-serving-cert\") pod \"service-ca-operator-544d749c8c-2tk8w\" (UID: \"195f952d-ee96-485d-99aa-8b47e6633c97\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2tk8w" Oct 30 17:12:23.215231 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215185 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-trusted-ca\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:23.215231 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215200 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/16cd937e-4c6d-45d0-a9de-f77dc836d082-snapshots\") pod \"insights-operator-7c4986d449-4w8fl\" (UID: \"16cd937e-4c6d-45d0-a9de-f77dc836d082\") " pod="openshift-insights/insights-operator-7c4986d449-4w8fl" Oct 30 17:12:23.215231 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215242 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs2g4\" (UniqueName: \"kubernetes.io/projected/c7b7edd7-11cc-4fac-92ae-a08140056adb-kube-api-access-xs2g4\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:12:23.215920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215267 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/27ac623c-2f57-44ec-a1b1-3ad87909452d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jccch\" (UID: \"27ac623c-2f57-44ec-a1b1-3ad87909452d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jccch" Oct 30 17:12:23.215920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215284 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/16cd937e-4c6d-45d0-a9de-f77dc836d082-snapshots\") pod \"insights-operator-7c4986d449-4w8fl\" (UID: \"16cd937e-4c6d-45d0-a9de-f77dc836d082\") " pod="openshift-insights/insights-operator-7c4986d449-4w8fl" Oct 30 17:12:23.215920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215292 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13444fae-8e71-41ee-8489-55b8606492fb-trusted-ca-bundle\") pod \"periodic-gathering-2ftd7-xc6xx\" (UID: \"13444fae-8e71-41ee-8489-55b8606492fb\") " pod="openshift-insights/periodic-gathering-2ftd7-xc6xx" Oct 30 17:12:23.215920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215315 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/195f952d-ee96-485d-99aa-8b47e6633c97-serving-cert\") pod \"service-ca-operator-544d749c8c-2tk8w\" (UID: \"195f952d-ee96-485d-99aa-8b47e6633c97\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2tk8w" Oct 30 17:12:23.215920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215337 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6084e9b4-6b79-46ab-91cf-8d292d1f4591-data-volume\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:23.215920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215364 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-default-certificate\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:23.215920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215392 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2939b3a2-1950-44c9-802c-a53861aa7aa8-metrics-tls\") pod \"dns-default-22vlj\" (UID: \"2939b3a2-1950-44c9-802c-a53861aa7aa8\") " pod="openshift-dns/dns-default-22vlj" Oct 30 17:12:23.215920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215419 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-run-ovn\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.215920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215451 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6084e9b4-6b79-46ab-91cf-8d292d1f4591-data-volume\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:23.215920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215461 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/27ac623c-2f57-44ec-a1b1-3ad87909452d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jccch\" (UID: \"27ac623c-2f57-44ec-a1b1-3ad87909452d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jccch" Oct 30 17:12:23.215920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215519 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-trusted-ca\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:23.215920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215564 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.215920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215595 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13444fae-8e71-41ee-8489-55b8606492fb-trusted-ca-bundle\") pod \"periodic-gathering-2ftd7-xc6xx\" (UID: \"13444fae-8e71-41ee-8489-55b8606492fb\") " pod="openshift-insights/periodic-gathering-2ftd7-xc6xx" Oct 30 17:12:23.215920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215612 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtlv4\" (UniqueName: \"kubernetes.io/projected/16cd937e-4c6d-45d0-a9de-f77dc836d082-kube-api-access-wtlv4\") pod \"insights-operator-7c4986d449-4w8fl\" (UID: \"16cd937e-4c6d-45d0-a9de-f77dc836d082\") " pod="openshift-insights/insights-operator-7c4986d449-4w8fl" Oct 30 17:12:23.215920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215652 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-run\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.215920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215679 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-var-lib-kubelet\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.215920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215720 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-default-certificate\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:23.215920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215748 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4bdc401b-ae3f-48c0-8110-8cd6096007d8-signing-key\") pod \"service-ca-5fffc97bdf-27cwx\" (UID: \"4bdc401b-ae3f-48c0-8110-8cd6096007d8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-27cwx" Oct 30 17:12:23.215920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215780 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-var-lib-openvswitch\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.215920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215821 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-image-registry-private-configuration\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:23.215920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215854 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l25lh\" (UniqueName: \"kubernetes.io/projected/195f952d-ee96-485d-99aa-8b47e6633c97-kube-api-access-l25lh\") pod \"service-ca-operator-544d749c8c-2tk8w\" (UID: \"195f952d-ee96-485d-99aa-8b47e6633c97\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2tk8w" Oct 30 17:12:23.215920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215893 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f442e60f-8ccf-485e-9709-60a909d482bc-konnectivity-ca\") pod \"konnectivity-agent-jnfks\" (UID: \"f442e60f-8ccf-485e-9709-60a909d482bc\") " pod="kube-system/konnectivity-agent-jnfks" Oct 30 17:12:23.215920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215908 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6178534d-9f75-49c7-a977-de65737cbcea-metrics-certs\") pod \"network-metrics-daemon-87jjv\" (UID: \"6178534d-9f75-49c7-a977-de65737cbcea\") " pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:12:23.215920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215923 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-kubernetes\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.216808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.215963 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kcdc\" (UniqueName: \"kubernetes.io/projected/c7ac81c7-e1f0-4549-886d-d7b8b893b614-kube-api-access-5kcdc\") pod \"kas-connection-checker-685895c7c9-9jx9m\" (UID: \"c7ac81c7-e1f0-4549-886d-d7b8b893b614\") " pod="kube-system/kas-connection-checker-685895c7c9-9jx9m" Oct 30 17:12:23.216808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.216009 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2664a881-44d7-470a-93d0-df1b3eaca0bc-host\") pod \"node-ca-stqrx\" (UID: \"2664a881-44d7-470a-93d0-df1b3eaca0bc\") " pod="openshift-image-registry/node-ca-stqrx" Oct 30 17:12:23.216808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.216033 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-image-registry-private-configuration\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:23.216808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.216029 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4bdc401b-ae3f-48c0-8110-8cd6096007d8-signing-key\") pod \"service-ca-5fffc97bdf-27cwx\" (UID: \"4bdc401b-ae3f-48c0-8110-8cd6096007d8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-27cwx" Oct 30 17:12:23.216808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.216059 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brgv8\" (UniqueName: \"kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-kube-api-access-brgv8\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:23.216808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.216090 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-metrics-certs\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:23.216808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.216104 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f442e60f-8ccf-485e-9709-60a909d482bc-konnectivity-ca\") pod \"konnectivity-agent-jnfks\" (UID: \"f442e60f-8ccf-485e-9709-60a909d482bc\") " pod="kube-system/konnectivity-agent-jnfks" Oct 30 17:12:23.216808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.216159 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csqbr\" (UniqueName: \"kubernetes.io/projected/2989c0bb-0ed6-41b2-ad3e-bb6e85bd60b9-kube-api-access-csqbr\") pod \"volume-data-source-validator-58c9f5f46d-4x66d\" (UID: \"2989c0bb-0ed6-41b2-ad3e-bb6e85bd60b9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4x66d" Oct 30 17:12:23.216808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.216211 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/13444fae-8e71-41ee-8489-55b8606492fb-archives-path\") pod \"periodic-gathering-2ftd7-xc6xx\" (UID: \"13444fae-8e71-41ee-8489-55b8606492fb\") " pod="openshift-insights/periodic-gathering-2ftd7-xc6xx" Oct 30 17:12:23.216808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.216260 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-sysconfig\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.216808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.216314 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/13444fae-8e71-41ee-8489-55b8606492fb-archives-path\") pod \"periodic-gathering-2ftd7-xc6xx\" (UID: \"13444fae-8e71-41ee-8489-55b8606492fb\") " pod="openshift-insights/periodic-gathering-2ftd7-xc6xx" Oct 30 17:12:23.216808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.216331 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-lib-modules\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.216808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.216370 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-var-lib-cni-multus\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.216808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.216390 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/933be7f4-8d62-4735-807f-7aa1439583fa-iptables-alerter-script\") pod \"iptables-alerter-pj5s7\" (UID: \"933be7f4-8d62-4735-807f-7aa1439583fa\") " pod="openshift-network-operator/iptables-alerter-pj5s7" Oct 30 17:12:23.216808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.216434 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-slash\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.216808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.216454 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d49vx\" (UniqueName: \"kubernetes.io/projected/f608efeb-1bda-4375-a428-c6db6341fd0b-kube-api-access-d49vx\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.216808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.216492 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9kmm\" (UniqueName: \"kubernetes.io/projected/1eab122e-07da-4307-b7f2-914cbe2b4237-kube-api-access-f9kmm\") pod \"console-operator-7cd56b8dbc-mf9rd\" (UID: \"1eab122e-07da-4307-b7f2-914cbe2b4237\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" Oct 30 17:12:23.216808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.216513 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-tls\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:23.216808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.216529 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd6ddeda-aec9-4872-9e43-084577cfcd7d-cnibin\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:12:23.216808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.216553 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r67qt\" (UniqueName: \"kubernetes.io/projected/cd6ddeda-aec9-4872-9e43-084577cfcd7d-kube-api-access-r67qt\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:12:23.216808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.216590 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-multus-socket-dir-parent\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.216808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.216684 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-run-netns\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.216808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.216694 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/933be7f4-8d62-4735-807f-7aa1439583fa-iptables-alerter-script\") pod \"iptables-alerter-pj5s7\" (UID: \"933be7f4-8d62-4735-807f-7aa1439583fa\") " pod="openshift-network-operator/iptables-alerter-pj5s7" Oct 30 17:12:23.216808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.216712 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2939b3a2-1950-44c9-802c-a53861aa7aa8-tmp-dir\") pod \"dns-default-22vlj\" (UID: \"2939b3a2-1950-44c9-802c-a53861aa7aa8\") " pod="openshift-dns/dns-default-22vlj" Oct 30 17:12:23.216808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.216767 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2939b3a2-1950-44c9-802c-a53861aa7aa8-tmp-dir\") pod \"dns-default-22vlj\" (UID: \"2939b3a2-1950-44c9-802c-a53861aa7aa8\") " pod="openshift-dns/dns-default-22vlj" Oct 30 17:12:23.216808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.216801 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-kubelet\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.217751 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.216825 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-etc-openvswitch\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.217751 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.216851 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rttdq\" (UniqueName: \"kubernetes.io/projected/cc71466d-eac0-460b-af50-3204f07b3f35-kube-api-access-rttdq\") pod \"cluster-monitoring-operator-884576c57-6jbgg\" (UID: \"cc71466d-eac0-460b-af50-3204f07b3f35\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jbgg" Oct 30 17:12:23.217751 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.216890 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/16cd937e-4c6d-45d0-a9de-f77dc836d082-serving-cert\") pod \"insights-operator-7c4986d449-4w8fl\" (UID: \"16cd937e-4c6d-45d0-a9de-f77dc836d082\") " pod="openshift-insights/insights-operator-7c4986d449-4w8fl" Oct 30 17:12:23.217751 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.216953 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2906e95e-3baa-4c79-ad9c-b2592c1d414f-dbus\") pod \"global-pull-secret-syncer-5mtjr\" (UID: \"2906e95e-3baa-4c79-ad9c-b2592c1d414f\") " pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:12:23.217751 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.217029 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6084e9b4-6b79-46ab-91cf-8d292d1f4591-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:23.217751 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.217065 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-multus-conf-dir\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.217751 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.217087 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4bdc401b-ae3f-48c0-8110-8cd6096007d8-signing-cabundle\") pod \"service-ca-5fffc97bdf-27cwx\" (UID: \"4bdc401b-ae3f-48c0-8110-8cd6096007d8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-27cwx" Oct 30 17:12:23.217751 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.217135 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffnbr\" (UniqueName: \"kubernetes.io/projected/2939b3a2-1950-44c9-802c-a53861aa7aa8-kube-api-access-ffnbr\") pod \"dns-default-22vlj\" (UID: \"2939b3a2-1950-44c9-802c-a53861aa7aa8\") " pod="openshift-dns/dns-default-22vlj" Oct 30 17:12:23.217751 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.217200 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/933be7f4-8d62-4735-807f-7aa1439583fa-host-slash\") pod \"iptables-alerter-pj5s7\" (UID: \"933be7f4-8d62-4735-807f-7aa1439583fa\") " pod="openshift-network-operator/iptables-alerter-pj5s7" Oct 30 17:12:23.217751 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.217243 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4bdc401b-ae3f-48c0-8110-8cd6096007d8-signing-cabundle\") pod \"service-ca-5fffc97bdf-27cwx\" (UID: \"4bdc401b-ae3f-48c0-8110-8cd6096007d8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-27cwx" Oct 30 17:12:23.217751 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.217271 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/16cd937e-4c6d-45d0-a9de-f77dc836d082-serving-cert\") pod \"insights-operator-7c4986d449-4w8fl\" (UID: \"16cd937e-4c6d-45d0-a9de-f77dc836d082\") " pod="openshift-insights/insights-operator-7c4986d449-4w8fl" Oct 30 17:12:23.217751 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.217302 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-var-lib-kubelet\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.217751 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.217364 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c2d3293d-1890-4fa2-b790-9f84622c9c8b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kvpw6\" (UID: \"c2d3293d-1890-4fa2-b790-9f84622c9c8b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kvpw6" Oct 30 17:12:23.217751 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.217383 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-multus-cni-dir\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.217751 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.217406 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-run-multus-certs\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.217751 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.217431 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sh5q\" (UniqueName: \"kubernetes.io/projected/933be7f4-8d62-4735-807f-7aa1439583fa-kube-api-access-6sh5q\") pod \"iptables-alerter-pj5s7\" (UID: \"933be7f4-8d62-4735-807f-7aa1439583fa\") " pod="openshift-network-operator/iptables-alerter-pj5s7" Oct 30 17:12:23.217751 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.217454 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1eab122e-07da-4307-b7f2-914cbe2b4237-config\") pod \"console-operator-7cd56b8dbc-mf9rd\" (UID: \"1eab122e-07da-4307-b7f2-914cbe2b4237\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" Oct 30 17:12:23.217751 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.217558 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd74ba94-86d0-4707-94e5-6822c4ab5751-ca-trust-extracted\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:23.217751 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.217629 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd6ddeda-aec9-4872-9e43-084577cfcd7d-cni-binary-copy\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:12:23.217751 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.217641 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1eab122e-07da-4307-b7f2-914cbe2b4237-config\") pod \"console-operator-7cd56b8dbc-mf9rd\" (UID: \"1eab122e-07da-4307-b7f2-914cbe2b4237\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" Oct 30 17:12:23.217751 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.217630 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd74ba94-86d0-4707-94e5-6822c4ab5751-ca-trust-extracted\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:23.217751 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.217651 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-socket-dir\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:12:23.217751 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.217685 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89eff971-c01b-4b06-9c16-4332956558a3-cert\") pod \"ingress-canary-rdjn5\" (UID: \"89eff971-c01b-4b06-9c16-4332956558a3\") " pod="openshift-ingress-canary/ingress-canary-rdjn5" Oct 30 17:12:23.217751 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.217723 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-log-socket\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.217751 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.217741 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-ca-trust-extracted\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:23.217751 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.217757 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd74ba94-86d0-4707-94e5-6822c4ab5751-installation-pull-secrets\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:23.217751 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.217773 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cc71466d-eac0-460b-af50-3204f07b3f35-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6jbgg\" (UID: \"cc71466d-eac0-460b-af50-3204f07b3f35\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jbgg" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.217788 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16cd937e-4c6d-45d0-a9de-f77dc836d082-tmp\") pod \"insights-operator-7c4986d449-4w8fl\" (UID: \"16cd937e-4c6d-45d0-a9de-f77dc836d082\") " pod="openshift-insights/insights-operator-7c4986d449-4w8fl" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.217810 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd6ddeda-aec9-4872-9e43-084577cfcd7d-cni-binary-copy\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.217811 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6084e9b4-6b79-46ab-91cf-8d292d1f4591-crio-socket\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.217834 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89eff971-c01b-4b06-9c16-4332956558a3-cert\") pod \"ingress-canary-rdjn5\" (UID: \"89eff971-c01b-4b06-9c16-4332956558a3\") " pod="openshift-ingress-canary/ingress-canary-rdjn5" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.217848 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-ca-trust-extracted\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.217903 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16cd937e-4c6d-45d0-a9de-f77dc836d082-tmp\") pod \"insights-operator-7c4986d449-4w8fl\" (UID: \"16cd937e-4c6d-45d0-a9de-f77dc836d082\") " pod="openshift-insights/insights-operator-7c4986d449-4w8fl" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.217990 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd74ba94-86d0-4707-94e5-6822c4ab5751-installation-pull-secrets\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218025 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5mmm\" (UniqueName: \"kubernetes.io/projected/2664a881-44d7-470a-93d0-df1b3eaca0bc-kube-api-access-p5mmm\") pod \"node-ca-stqrx\" (UID: \"2664a881-44d7-470a-93d0-df1b3eaca0bc\") " pod="openshift-image-registry/node-ca-stqrx" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218046 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-run-netns\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218066 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2906e95e-3baa-4c79-ad9c-b2592c1d414f-original-pull-secret\") pod \"global-pull-secret-syncer-5mtjr\" (UID: \"2906e95e-3baa-4c79-ad9c-b2592c1d414f\") " pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218082 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218124 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cc71466d-eac0-460b-af50-3204f07b3f35-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6jbgg\" (UID: \"cc71466d-eac0-460b-af50-3204f07b3f35\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jbgg" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218152 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f608efeb-1bda-4375-a428-c6db6341fd0b-ovn-node-metrics-cert\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218196 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd6ddeda-aec9-4872-9e43-084577cfcd7d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218230 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-cnibin\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218285 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1eab122e-07da-4307-b7f2-914cbe2b4237-serving-cert\") pod \"console-operator-7cd56b8dbc-mf9rd\" (UID: \"1eab122e-07da-4307-b7f2-914cbe2b4237\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218305 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2906e95e-3baa-4c79-ad9c-b2592c1d414f-original-pull-secret\") pod \"global-pull-secret-syncer-5mtjr\" (UID: \"2906e95e-3baa-4c79-ad9c-b2592c1d414f\") " pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218309 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16cd937e-4c6d-45d0-a9de-f77dc836d082-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4w8fl\" (UID: \"16cd937e-4c6d-45d0-a9de-f77dc836d082\") " pod="openshift-insights/insights-operator-7c4986d449-4w8fl" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218328 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-var-lib-cni-bin\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218344 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/27ac623c-2f57-44ec-a1b1-3ad87909452d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jccch\" (UID: \"27ac623c-2f57-44ec-a1b1-3ad87909452d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jccch" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218360 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txhrv\" (UniqueName: \"kubernetes.io/projected/03a0e7f9-9e86-4aa9-9864-205263f6a486-kube-api-access-txhrv\") pod \"kas-connection-checker-685895c7c9-w5rnm\" (UID: \"03a0e7f9-9e86-4aa9-9864-205263f6a486\") " pod="kube-system/kas-connection-checker-685895c7c9-w5rnm" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218375 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-bound-sa-token\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218393 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d05e04a8-3ddf-4261-add0-c2324c6c9e21-tmp\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218442 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9swv\" (UniqueName: \"kubernetes.io/projected/55f62602-c7b7-4a6c-b244-f6e48775733f-kube-api-access-m9swv\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218487 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1eab122e-07da-4307-b7f2-914cbe2b4237-serving-cert\") pod \"console-operator-7cd56b8dbc-mf9rd\" (UID: \"1eab122e-07da-4307-b7f2-914cbe2b4237\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218493 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f608efeb-1bda-4375-a428-c6db6341fd0b-ovn-node-metrics-cert\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218525 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-sys-fs\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218528 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/27ac623c-2f57-44ec-a1b1-3ad87909452d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jccch\" (UID: \"27ac623c-2f57-44ec-a1b1-3ad87909452d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jccch" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218557 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-sysctl-conf\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218583 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-run-openvswitch\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218530 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d05e04a8-3ddf-4261-add0-c2324c6c9e21-tmp\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218636 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16cd937e-4c6d-45d0-a9de-f77dc836d082-service-ca-bundle\") pod \"insights-operator-7c4986d449-4w8fl\" (UID: \"16cd937e-4c6d-45d0-a9de-f77dc836d082\") " pod="openshift-insights/insights-operator-7c4986d449-4w8fl" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218646 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16cd937e-4c6d-45d0-a9de-f77dc836d082-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4w8fl\" (UID: \"16cd937e-4c6d-45d0-a9de-f77dc836d082\") " pod="openshift-insights/insights-operator-7c4986d449-4w8fl" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218664 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/55f62602-c7b7-4a6c-b244-f6e48775733f-multus-daemon-config\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218682 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-registration-dir\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:12:23.218747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218737 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2664a881-44d7-470a-93d0-df1b3eaca0bc-serviceca\") pod \"node-ca-stqrx\" (UID: \"2664a881-44d7-470a-93d0-df1b3eaca0bc\") " pod="openshift-image-registry/node-ca-stqrx" Oct 30 17:12:23.219901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218775 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f608efeb-1bda-4375-a428-c6db6341fd0b-env-overrides\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.219901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218795 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-sysctl-d\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.219901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218812 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16cd937e-4c6d-45d0-a9de-f77dc836d082-service-ca-bundle\") pod \"insights-operator-7c4986d449-4w8fl\" (UID: \"16cd937e-4c6d-45d0-a9de-f77dc836d082\") " pod="openshift-insights/insights-operator-7c4986d449-4w8fl" Oct 30 17:12:23.219901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218884 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/55f62602-c7b7-4a6c-b244-f6e48775733f-multus-daemon-config\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.219901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218908 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2906e95e-3baa-4c79-ad9c-b2592c1d414f-kubelet-config\") pod \"global-pull-secret-syncer-5mtjr\" (UID: \"2906e95e-3baa-4c79-ad9c-b2592c1d414f\") " pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:12:23.219901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218947 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddss9\" (UniqueName: \"kubernetes.io/projected/89eff971-c01b-4b06-9c16-4332956558a3-kube-api-access-ddss9\") pod \"ingress-canary-rdjn5\" (UID: \"89eff971-c01b-4b06-9c16-4332956558a3\") " pod="openshift-ingress-canary/ingress-canary-rdjn5" Oct 30 17:12:23.219901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.218983 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f608efeb-1bda-4375-a428-c6db6341fd0b-env-overrides\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.219901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.219000 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2939b3a2-1950-44c9-802c-a53861aa7aa8-config-volume\") pod \"dns-default-22vlj\" (UID: \"2939b3a2-1950-44c9-802c-a53861aa7aa8\") " pod="openshift-dns/dns-default-22vlj" Oct 30 17:12:23.219901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.219037 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-cni-bin\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.219901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.219064 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f608efeb-1bda-4375-a428-c6db6341fd0b-ovnkube-config\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.219901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.219105 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-tls\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:23.219901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.219168 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2939b3a2-1950-44c9-802c-a53861aa7aa8-config-volume\") pod \"dns-default-22vlj\" (UID: \"2939b3a2-1950-44c9-802c-a53861aa7aa8\") " pod="openshift-dns/dns-default-22vlj" Oct 30 17:12:23.219901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.219213 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-systemd\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.219901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.219253 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-system-cni-dir\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.219901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.219291 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/745f4f0f-74f7-4fff-9384-c8f45f601e33-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-t9hdn\" (UID: \"745f4f0f-74f7-4fff-9384-c8f45f601e33\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t9hdn" Oct 30 17:12:23.219901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.219308 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f608efeb-1bda-4375-a428-c6db6341fd0b-ovnkube-config\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.219901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.219332 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-certificates\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:23.219901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.219357 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd74ba94-86d0-4707-94e5-6822c4ab5751-image-registry-private-configuration\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:23.219901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.219388 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2664a881-44d7-470a-93d0-df1b3eaca0bc-serviceca\") pod \"node-ca-stqrx\" (UID: \"2664a881-44d7-470a-93d0-df1b3eaca0bc\") " pod="openshift-image-registry/node-ca-stqrx" Oct 30 17:12:23.219901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.219401 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-stats-auth\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:23.219901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.219507 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-run-systemd\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.219901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.219551 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd74ba94-86d0-4707-94e5-6822c4ab5751-image-registry-private-configuration\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:23.219901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.219585 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-run-ovn-kubernetes\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.219901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.219705 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-stats-auth\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:23.220517 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.219947 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-certificates\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:23.239700 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.239681 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztbbl\" (UniqueName: \"kubernetes.io/projected/745f4f0f-74f7-4fff-9384-c8f45f601e33-kube-api-access-ztbbl\") pod \"cluster-samples-operator-6ddb6c8b94-t9hdn\" (UID: \"745f4f0f-74f7-4fff-9384-c8f45f601e33\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t9hdn" Oct 30 17:12:23.239778 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.239757 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtlv4\" (UniqueName: \"kubernetes.io/projected/16cd937e-4c6d-45d0-a9de-f77dc836d082-kube-api-access-wtlv4\") pod \"insights-operator-7c4986d449-4w8fl\" (UID: \"16cd937e-4c6d-45d0-a9de-f77dc836d082\") " pod="openshift-insights/insights-operator-7c4986d449-4w8fl" Oct 30 17:12:23.240438 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.240409 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zxfh\" (UniqueName: \"kubernetes.io/projected/6084e9b4-6b79-46ab-91cf-8d292d1f4591-kube-api-access-9zxfh\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:23.240643 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.240621 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7n64\" (UniqueName: \"kubernetes.io/projected/467187ee-7039-4bb4-adbe-38161ee7c336-kube-api-access-p7n64\") pod \"node-resolver-zgb2b\" (UID: \"467187ee-7039-4bb4-adbe-38161ee7c336\") " pod="openshift-dns/node-resolver-zgb2b" Oct 30 17:12:23.240738 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.240717 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4qm2\" (UniqueName: \"kubernetes.io/projected/27ac623c-2f57-44ec-a1b1-3ad87909452d-kube-api-access-g4qm2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jccch\" (UID: \"27ac623c-2f57-44ec-a1b1-3ad87909452d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jccch" Oct 30 17:12:23.240738 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.240732 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dz8c\" (UniqueName: \"kubernetes.io/projected/e30e6b59-b8e2-4125-ae7a-41edd1ce7389-kube-api-access-5dz8c\") pod \"kas-connection-checker-685895c7c9-hwfbh\" (UID: \"e30e6b59-b8e2-4125-ae7a-41edd1ce7389\") " pod="kube-system/kas-connection-checker-685895c7c9-hwfbh" Oct 30 17:12:23.240834 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.240813 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs2g4\" (UniqueName: \"kubernetes.io/projected/c7b7edd7-11cc-4fac-92ae-a08140056adb-kube-api-access-xs2g4\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:12:23.240834 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.240829 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drkzm\" (UniqueName: \"kubernetes.io/projected/13444fae-8e71-41ee-8489-55b8606492fb-kube-api-access-drkzm\") pod \"periodic-gathering-2ftd7-xc6xx\" (UID: \"13444fae-8e71-41ee-8489-55b8606492fb\") " pod="openshift-insights/periodic-gathering-2ftd7-xc6xx" Oct 30 17:12:23.240917 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.240849 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrqrz\" (UniqueName: \"kubernetes.io/projected/924ac066-ff98-4320-9eb2-268f0a39ca33-kube-api-access-mrqrz\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:23.241697 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.241679 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-bound-sa-token\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:23.242477 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.242461 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brgv8\" (UniqueName: \"kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-kube-api-access-brgv8\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:23.245199 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.245155 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l25lh\" (UniqueName: \"kubernetes.io/projected/195f952d-ee96-485d-99aa-8b47e6633c97-kube-api-access-l25lh\") pod \"service-ca-operator-544d749c8c-2tk8w\" (UID: \"195f952d-ee96-485d-99aa-8b47e6633c97\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2tk8w" Oct 30 17:12:23.245291 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.245259 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx4rg\" (UniqueName: \"kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-kube-api-access-qx4rg\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:23.245424 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.245405 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4njs\" (UniqueName: \"kubernetes.io/projected/d39ddd3a-781f-4084-b4e2-624b130c1025-kube-api-access-z4njs\") pod \"network-check-target-t2pqm\" (UID: \"d39ddd3a-781f-4084-b4e2-624b130c1025\") " pod="openshift-network-diagnostics/network-check-target-t2pqm" Oct 30 17:12:23.245592 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.245570 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkvxq\" (UniqueName: \"kubernetes.io/projected/4bdc401b-ae3f-48c0-8110-8cd6096007d8-kube-api-access-xkvxq\") pod \"service-ca-5fffc97bdf-27cwx\" (UID: \"4bdc401b-ae3f-48c0-8110-8cd6096007d8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-27cwx" Oct 30 17:12:23.245693 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.245681 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csqbr\" (UniqueName: \"kubernetes.io/projected/2989c0bb-0ed6-41b2-ad3e-bb6e85bd60b9-kube-api-access-csqbr\") pod \"volume-data-source-validator-58c9f5f46d-4x66d\" (UID: \"2989c0bb-0ed6-41b2-ad3e-bb6e85bd60b9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4x66d" Oct 30 17:12:23.245852 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.245834 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffnbr\" (UniqueName: \"kubernetes.io/projected/2939b3a2-1950-44c9-802c-a53861aa7aa8-kube-api-access-ffnbr\") pod \"dns-default-22vlj\" (UID: \"2939b3a2-1950-44c9-802c-a53861aa7aa8\") " pod="openshift-dns/dns-default-22vlj" Oct 30 17:12:23.245955 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.245935 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-bound-sa-token\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:23.246012 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.245975 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txhrv\" (UniqueName: \"kubernetes.io/projected/03a0e7f9-9e86-4aa9-9864-205263f6a486-kube-api-access-txhrv\") pod \"kas-connection-checker-685895c7c9-w5rnm\" (UID: \"03a0e7f9-9e86-4aa9-9864-205263f6a486\") " pod="kube-system/kas-connection-checker-685895c7c9-w5rnm" Oct 30 17:12:23.246074 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.246048 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r67qt\" (UniqueName: \"kubernetes.io/projected/cd6ddeda-aec9-4872-9e43-084577cfcd7d-kube-api-access-r67qt\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:12:23.246628 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.246601 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d49vx\" (UniqueName: \"kubernetes.io/projected/f608efeb-1bda-4375-a428-c6db6341fd0b-kube-api-access-d49vx\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.247179 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.247157 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sh5q\" (UniqueName: \"kubernetes.io/projected/933be7f4-8d62-4735-807f-7aa1439583fa-kube-api-access-6sh5q\") pod \"iptables-alerter-pj5s7\" (UID: \"933be7f4-8d62-4735-807f-7aa1439583fa\") " pod="openshift-network-operator/iptables-alerter-pj5s7" Oct 30 17:12:23.247467 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.247449 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kcdc\" (UniqueName: \"kubernetes.io/projected/c7ac81c7-e1f0-4549-886d-d7b8b893b614-kube-api-access-5kcdc\") pod \"kas-connection-checker-685895c7c9-9jx9m\" (UID: \"c7ac81c7-e1f0-4549-886d-d7b8b893b614\") " pod="kube-system/kas-connection-checker-685895c7c9-9jx9m" Oct 30 17:12:23.248006 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.247991 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddss9\" (UniqueName: \"kubernetes.io/projected/89eff971-c01b-4b06-9c16-4332956558a3-kube-api-access-ddss9\") pod \"ingress-canary-rdjn5\" (UID: \"89eff971-c01b-4b06-9c16-4332956558a3\") " pod="openshift-ingress-canary/ingress-canary-rdjn5" Oct 30 17:12:23.249715 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.249692 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5mmm\" (UniqueName: \"kubernetes.io/projected/2664a881-44d7-470a-93d0-df1b3eaca0bc-kube-api-access-p5mmm\") pod \"node-ca-stqrx\" (UID: \"2664a881-44d7-470a-93d0-df1b3eaca0bc\") " pod="openshift-image-registry/node-ca-stqrx" Oct 30 17:12:23.250325 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.250312 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9kmm\" (UniqueName: \"kubernetes.io/projected/1eab122e-07da-4307-b7f2-914cbe2b4237-kube-api-access-f9kmm\") pod \"console-operator-7cd56b8dbc-mf9rd\" (UID: \"1eab122e-07da-4307-b7f2-914cbe2b4237\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" Oct 30 17:12:23.252001 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.251979 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqrt8\" (UniqueName: \"kubernetes.io/projected/6178534d-9f75-49c7-a977-de65737cbcea-kube-api-access-nqrt8\") pod \"network-metrics-daemon-87jjv\" (UID: \"6178534d-9f75-49c7-a977-de65737cbcea\") " pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:12:23.252062 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.252049 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn8lc\" (UniqueName: \"kubernetes.io/projected/d05e04a8-3ddf-4261-add0-c2324c6c9e21-kube-api-access-jn8lc\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.252257 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.252243 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rttdq\" (UniqueName: \"kubernetes.io/projected/cc71466d-eac0-460b-af50-3204f07b3f35-kube-api-access-rttdq\") pod \"cluster-monitoring-operator-884576c57-6jbgg\" (UID: \"cc71466d-eac0-460b-af50-3204f07b3f35\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jbgg" Oct 30 17:12:23.252314 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.252299 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zz48\" (UniqueName: \"kubernetes.io/projected/e5f38da3-fe1f-4f36-8ed3-12cb01bf128b-kube-api-access-5zz48\") pod \"network-check-source-6f78d59f99-z6b8h\" (UID: \"e5f38da3-fe1f-4f36-8ed3-12cb01bf128b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z6b8h" Oct 30 17:12:23.257623 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.257606 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9swv\" (UniqueName: \"kubernetes.io/projected/55f62602-c7b7-4a6c-b244-f6e48775733f-kube-api-access-m9swv\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.320318 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320269 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-metrics-certs\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:23.320318 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320296 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-sysconfig\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.320318 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320312 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-lib-modules\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.320531 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320327 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-var-lib-cni-multus\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.320531 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320344 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-slash\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.320531 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320359 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-tls\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:23.320531 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320381 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd6ddeda-aec9-4872-9e43-084577cfcd7d-cnibin\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:12:23.320531 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320415 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-multus-socket-dir-parent\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.320531 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320424 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-slash\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.320531 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320437 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-run-netns\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.320531 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320420 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-sysconfig\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.320531 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320452 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-var-lib-cni-multus\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.320531 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320466 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-kubelet\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.320531 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320474 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd6ddeda-aec9-4872-9e43-084577cfcd7d-cnibin\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:12:23.320531 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320473 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-lib-modules\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.320531 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:23.320492 4965 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 17:12:23.320531 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320503 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-etc-openvswitch\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.320531 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320517 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-run-netns\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.320531 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320534 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-etc-openvswitch\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.321232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320535 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-multus-socket-dir-parent\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.321232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320507 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-kubelet\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.321232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320571 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2906e95e-3baa-4c79-ad9c-b2592c1d414f-dbus\") pod \"global-pull-secret-syncer-5mtjr\" (UID: \"2906e95e-3baa-4c79-ad9c-b2592c1d414f\") " pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:12:23.321232 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:23.320601 4965 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-metrics-certs podName:924ac066-ff98-4320-9eb2-268f0a39ca33 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:23.820581446 +0000 UTC m=+1.872681372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-metrics-certs") pod "router-default-ddb54c4f7-54tw2" (UID: "924ac066-ff98-4320-9eb2-268f0a39ca33") : secret "router-metrics-certs-default" not found Oct 30 17:12:23.321232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320637 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6084e9b4-6b79-46ab-91cf-8d292d1f4591-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:23.321232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320661 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-multus-conf-dir\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.321232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320703 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/933be7f4-8d62-4735-807f-7aa1439583fa-host-slash\") pod \"iptables-alerter-pj5s7\" (UID: \"933be7f4-8d62-4735-807f-7aa1439583fa\") " pod="openshift-network-operator/iptables-alerter-pj5s7" Oct 30 17:12:23.321232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320638 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2906e95e-3baa-4c79-ad9c-b2592c1d414f-dbus\") pod \"global-pull-secret-syncer-5mtjr\" (UID: \"2906e95e-3baa-4c79-ad9c-b2592c1d414f\") " pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:12:23.321232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320721 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-multus-conf-dir\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.321232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320741 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-var-lib-kubelet\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.321232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320774 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/933be7f4-8d62-4735-807f-7aa1439583fa-host-slash\") pod \"iptables-alerter-pj5s7\" (UID: \"933be7f4-8d62-4735-807f-7aa1439583fa\") " pod="openshift-network-operator/iptables-alerter-pj5s7" Oct 30 17:12:23.321232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320765 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c2d3293d-1890-4fa2-b790-9f84622c9c8b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kvpw6\" (UID: \"c2d3293d-1890-4fa2-b790-9f84622c9c8b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kvpw6" Oct 30 17:12:23.321232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320780 4965 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 17:12:23.321232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320813 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-var-lib-kubelet\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.321232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320817 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-multus-cni-dir\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.321232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320841 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-run-multus-certs\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.321232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320969 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-socket-dir\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:12:23.321232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320980 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-run-multus-certs\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.321232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.320994 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-log-socket\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.321232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321050 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-multus-cni-dir\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.321232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321038 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6084e9b4-6b79-46ab-91cf-8d292d1f4591-crio-socket\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:23.321232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321084 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-socket-dir\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:12:23.321232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321095 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-run-netns\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.321232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321099 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6084e9b4-6b79-46ab-91cf-8d292d1f4591-crio-socket\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:23.321232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321122 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:12:23.321232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321129 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-log-socket\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.321232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321146 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-run-netns\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.321232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321148 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd6ddeda-aec9-4872-9e43-084577cfcd7d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:12:23.321232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321158 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6084e9b4-6b79-46ab-91cf-8d292d1f4591-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:23.321232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321172 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:12:23.321232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321199 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd6ddeda-aec9-4872-9e43-084577cfcd7d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:12:23.321232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321226 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-cnibin\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321260 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-var-lib-cni-bin\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321283 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-sys-fs\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321313 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-sysctl-conf\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321325 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-var-lib-cni-bin\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321391 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-sys-fs\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321342 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-cnibin\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321364 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-run-openvswitch\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321335 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-run-openvswitch\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321458 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-registration-dir\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321486 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-sysctl-d\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321507 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2906e95e-3baa-4c79-ad9c-b2592c1d414f-kubelet-config\") pod \"global-pull-secret-syncer-5mtjr\" (UID: \"2906e95e-3baa-4c79-ad9c-b2592c1d414f\") " pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321511 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-sysctl-conf\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321570 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-registration-dir\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321605 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-cni-bin\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321574 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-cni-bin\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321646 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-tls\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321649 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2906e95e-3baa-4c79-ad9c-b2592c1d414f-kubelet-config\") pod \"global-pull-secret-syncer-5mtjr\" (UID: \"2906e95e-3baa-4c79-ad9c-b2592c1d414f\") " pod="kube-system/global-pull-secret-syncer-5mtjr" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321635 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-sysctl-d\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321757 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-systemd\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321823 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-systemd\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.321920 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-system-cni-dir\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.322241 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-system-cni-dir\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.322311 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/745f4f0f-74f7-4fff-9384-c8f45f601e33-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-t9hdn\" (UID: \"745f4f0f-74f7-4fff-9384-c8f45f601e33\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t9hdn" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.322344 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-run-systemd\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.322370 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-run-ovn-kubernetes\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.322399 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc71466d-eac0-460b-af50-3204f07b3f35-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6jbgg\" (UID: \"cc71466d-eac0-460b-af50-3204f07b3f35\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jbgg" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.322426 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6084e9b4-6b79-46ab-91cf-8d292d1f4591-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.322434 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-run-systemd\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.322450 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-systemd-units\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.322481 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-modprobe-d\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.322508 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/924ac066-ff98-4320-9eb2-268f0a39ca33-service-ca-bundle\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.322534 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-etc-selinux\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:23.322544 4965 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.322556 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-node-log\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:23.322565 4965 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.322584 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-hostroot\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.322612 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-etc-kubernetes\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:23.322647 4965 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc71466d-eac0-460b-af50-3204f07b3f35-cluster-monitoring-operator-tls podName:cc71466d-eac0-460b-af50-3204f07b3f35 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:23.822606362 +0000 UTC m=+1.874706287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc71466d-eac0-460b-af50-3204f07b3f35-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6jbgg" (UID: "cc71466d-eac0-460b-af50-3204f07b3f35") : secret "cluster-monitoring-operator-tls" not found Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.322664 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-etc-kubernetes\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.322684 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd6ddeda-aec9-4872-9e43-084577cfcd7d-system-cni-dir\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.322703 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-systemd-units\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.322692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.322440 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-run-ovn-kubernetes\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.324470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.322726 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-host\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.324470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.322750 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-run-k8s-cni-cncf-io\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.324470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.322818 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-modprobe-d\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.324470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.322880 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-etc-selinux\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:12:23.324470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.322881 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-node-log\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.324470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.322935 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-hostroot\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.324470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.323457 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-tls\") pod \"image-registry-77579944f4-hctng\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:23.324470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.323502 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/924ac066-ff98-4320-9eb2-268f0a39ca33-service-ca-bundle\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:23.324470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.323555 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd6ddeda-aec9-4872-9e43-084577cfcd7d-system-cni-dir\") pod \"multus-additional-cni-plugins-7gvrp\" (UID: \"cd6ddeda-aec9-4872-9e43-084577cfcd7d\") " pod="openshift-multus/multus-additional-cni-plugins-7gvrp" Oct 30 17:12:23.324470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.323566 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-host\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.324470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.323578 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c2d3293d-1890-4fa2-b790-9f84622c9c8b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kvpw6\" (UID: \"c2d3293d-1890-4fa2-b790-9f84622c9c8b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kvpw6" Oct 30 17:12:23.324470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.323607 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-run-k8s-cni-cncf-io\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.324470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.323635 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-sys\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.324470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.323672 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-device-dir\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:12:23.324470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.323693 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-cni-netd\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.324470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.323707 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-sys\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.324470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.323761 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/467187ee-7039-4bb4-adbe-38161ee7c336-hosts-file\") pod \"node-resolver-zgb2b\" (UID: \"467187ee-7039-4bb4-adbe-38161ee7c336\") " pod="openshift-dns/node-resolver-zgb2b" Oct 30 17:12:23.324470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.323807 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2939b3a2-1950-44c9-802c-a53861aa7aa8-metrics-tls\") pod \"dns-default-22vlj\" (UID: \"2939b3a2-1950-44c9-802c-a53861aa7aa8\") " pod="openshift-dns/dns-default-22vlj" Oct 30 17:12:23.324470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.323810 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c7b7edd7-11cc-4fac-92ae-a08140056adb-device-dir\") pod \"aws-ebs-csi-driver-node-rvpvq\" (UID: \"c7b7edd7-11cc-4fac-92ae-a08140056adb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvpvq" Oct 30 17:12:23.324470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.323830 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-cni-netd\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.324470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.323830 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-run-ovn\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.324470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.323850 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/467187ee-7039-4bb4-adbe-38161ee7c336-hosts-file\") pod \"node-resolver-zgb2b\" (UID: \"467187ee-7039-4bb4-adbe-38161ee7c336\") " pod="openshift-dns/node-resolver-zgb2b" Oct 30 17:12:23.324470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.323905 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-run-ovn\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.324470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.323878 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.324470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.323922 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.324470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.323943 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-run\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.324470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.323966 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-var-lib-kubelet\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.324470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.323984 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-var-lib-openvswitch\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.324470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.324001 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6178534d-9f75-49c7-a977-de65737cbcea-metrics-certs\") pod \"network-metrics-daemon-87jjv\" (UID: \"6178534d-9f75-49c7-a977-de65737cbcea\") " pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:12:23.324470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.324036 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-kubernetes\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.324470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.324029 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-run\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.324470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.324153 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2664a881-44d7-470a-93d0-df1b3eaca0bc-host\") pod \"node-ca-stqrx\" (UID: \"2664a881-44d7-470a-93d0-df1b3eaca0bc\") " pod="openshift-image-registry/node-ca-stqrx" Oct 30 17:12:23.325879 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.324648 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2664a881-44d7-470a-93d0-df1b3eaca0bc-host\") pod \"node-ca-stqrx\" (UID: \"2664a881-44d7-470a-93d0-df1b3eaca0bc\") " pod="openshift-image-registry/node-ca-stqrx" Oct 30 17:12:23.325879 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:23.324731 4965 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:12:23.325879 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.324774 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55f62602-c7b7-4a6c-b244-f6e48775733f-host-var-lib-kubelet\") pod \"multus-db6jk\" (UID: \"55f62602-c7b7-4a6c-b244-f6e48775733f\") " pod="openshift-multus/multus-db6jk" Oct 30 17:12:23.325879 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.324779 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f608efeb-1bda-4375-a428-c6db6341fd0b-var-lib-openvswitch\") pod \"ovnkube-node-w54kt\" (UID: \"f608efeb-1bda-4375-a428-c6db6341fd0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:23.325879 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:23.324807 4965 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6084e9b4-6b79-46ab-91cf-8d292d1f4591-insights-runtime-extractor-tls podName:6084e9b4-6b79-46ab-91cf-8d292d1f4591 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:23.824774626 +0000 UTC m=+1.876874548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6084e9b4-6b79-46ab-91cf-8d292d1f4591-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbqhc" (UID: "6084e9b4-6b79-46ab-91cf-8d292d1f4591") : secret "insights-runtime-extractor-tls" not found Oct 30 17:12:23.325879 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:23.324846 4965 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6178534d-9f75-49c7-a977-de65737cbcea-metrics-certs podName:6178534d-9f75-49c7-a977-de65737cbcea nodeName:}" failed. No retries permitted until 2026-10-30 17:12:23.824821118 +0000 UTC m=+1.876921032 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6178534d-9f75-49c7-a977-de65737cbcea-metrics-certs") pod "network-metrics-daemon-87jjv" (UID: "6178534d-9f75-49c7-a977-de65737cbcea") : secret "metrics-daemon-secret" not found Oct 30 17:12:23.325879 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.324972 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d05e04a8-3ddf-4261-add0-c2324c6c9e21-etc-kubernetes\") pod \"tuned-pvpvp\" (UID: \"d05e04a8-3ddf-4261-add0-c2324c6c9e21\") " pod="openshift-cluster-node-tuning-operator/tuned-pvpvp" Oct 30 17:12:23.325879 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.325292 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/745f4f0f-74f7-4fff-9384-c8f45f601e33-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-t9hdn\" (UID: \"745f4f0f-74f7-4fff-9384-c8f45f601e33\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t9hdn" Oct 30 17:12:23.325879 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.325296 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-tls\") pod \"image-registry-585b9dc6c5-dblkr\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:23.330511 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.330488 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2939b3a2-1950-44c9-802c-a53861aa7aa8-metrics-tls\") pod \"dns-default-22vlj\" (UID: \"2939b3a2-1950-44c9-802c-a53861aa7aa8\") " pod="openshift-dns/dns-default-22vlj" Oct 30 17:12:23.389858 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.389837 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t9hdn" Oct 30 17:12:23.390965 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.390937 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:23.391066 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.391002 4965 scope.go:122] "RemoveContainer" containerID="c562428fe21e43cf22e26ebeba52550c57e2741dc839171b41e59c2d589d4922" Oct 30 17:12:23.391495 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.391479 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-kvpw6" Oct 30 17:12:23.401058 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.401039 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:23.402091 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.402071 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-22vlj" Oct 30 17:12:23.712060 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.712031 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t9hdn"] Oct 30 17:12:23.757844 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.757820 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-585b9dc6c5-dblkr"] Oct 30 17:12:23.760729 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:23.760693 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb69bb7ff_4506_40a7_9eb8_d2f4d4b976ce.slice/crio-1db9d57d93616d600cb967e4161c6118f8991004bdfda9c4b38a8f706776ba0d WatchSource:0}: Error finding container 1db9d57d93616d600cb967e4161c6118f8991004bdfda9c4b38a8f706776ba0d: Status 404 returned error can't find the container with id 1db9d57d93616d600cb967e4161c6118f8991004bdfda9c4b38a8f706776ba0d Oct 30 17:12:23.778463 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.778445 4965 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:12:23.803045 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.803023 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-22vlj"] Oct 30 17:12:23.805981 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:23.805959 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2939b3a2_1950_44c9_802c_a53861aa7aa8.slice/crio-3ba04c06cf73d36bd12eb8870289633020741c397ca5f12b7bc6de365ade792a WatchSource:0}: Error finding container 3ba04c06cf73d36bd12eb8870289633020741c397ca5f12b7bc6de365ade792a: Status 404 returned error can't find the container with id 3ba04c06cf73d36bd12eb8870289633020741c397ca5f12b7bc6de365ade792a Oct 30 17:12:23.830810 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.830782 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6178534d-9f75-49c7-a977-de65737cbcea-metrics-certs\") pod \"network-metrics-daemon-87jjv\" (UID: \"6178534d-9f75-49c7-a977-de65737cbcea\") " pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:12:23.830928 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.830828 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-metrics-certs\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:23.830988 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:23.830946 4965 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:12:23.830988 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.830958 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc71466d-eac0-460b-af50-3204f07b3f35-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6jbgg\" (UID: \"cc71466d-eac0-460b-af50-3204f07b3f35\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jbgg" Oct 30 17:12:23.831106 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.830991 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6084e9b4-6b79-46ab-91cf-8d292d1f4591-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:23.831106 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:23.831007 4965 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6178534d-9f75-49c7-a977-de65737cbcea-metrics-certs podName:6178534d-9f75-49c7-a977-de65737cbcea nodeName:}" failed. No retries permitted until 2026-10-30 17:12:24.830989851 +0000 UTC m=+2.883089780 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6178534d-9f75-49c7-a977-de65737cbcea-metrics-certs") pod "network-metrics-daemon-87jjv" (UID: "6178534d-9f75-49c7-a977-de65737cbcea") : secret "metrics-daemon-secret" not found Oct 30 17:12:23.831106 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:23.831068 4965 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:12:23.831106 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:23.831085 4965 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 17:12:23.831272 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:23.831119 4965 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6084e9b4-6b79-46ab-91cf-8d292d1f4591-insights-runtime-extractor-tls podName:6084e9b4-6b79-46ab-91cf-8d292d1f4591 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:24.831102602 +0000 UTC m=+2.883202511 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6084e9b4-6b79-46ab-91cf-8d292d1f4591-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbqhc" (UID: "6084e9b4-6b79-46ab-91cf-8d292d1f4591") : secret "insights-runtime-extractor-tls" not found Oct 30 17:12:23.831272 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:23.831139 4965 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc71466d-eac0-460b-af50-3204f07b3f35-cluster-monitoring-operator-tls podName:cc71466d-eac0-460b-af50-3204f07b3f35 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:24.831129397 +0000 UTC m=+2.883229307 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc71466d-eac0-460b-af50-3204f07b3f35-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6jbgg" (UID: "cc71466d-eac0-460b-af50-3204f07b3f35") : secret "cluster-monitoring-operator-tls" not found Oct 30 17:12:23.831272 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:23.831010 4965 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 17:12:23.831272 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:23.831208 4965 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-metrics-certs podName:924ac066-ff98-4320-9eb2-268f0a39ca33 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:24.831192167 +0000 UTC m=+2.883292078 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-metrics-certs") pod "router-default-ddb54c4f7-54tw2" (UID: "924ac066-ff98-4320-9eb2-268f0a39ca33") : secret "router-metrics-certs-default" not found Oct 30 17:12:23.831272 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.831219 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-kvpw6"] Oct 30 17:12:23.834326 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:23.834277 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d3293d_1890_4fa2_b790_9f84622c9c8b.slice/crio-3e090e3fe9da270184bc8d6fdac9174f2dd56dc6b4e29fdd4653ff273fa6d479 WatchSource:0}: Error finding container 3e090e3fe9da270184bc8d6fdac9174f2dd56dc6b4e29fdd4653ff273fa6d479: Status 404 returned error can't find the container with id 3e090e3fe9da270184bc8d6fdac9174f2dd56dc6b4e29fdd4653ff273fa6d479 Oct 30 17:12:23.844345 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:23.844314 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77579944f4-hctng"] Oct 30 17:12:23.847060 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:23.847038 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd74ba94_86d0_4707_94e5_6822c4ab5751.slice/crio-160da2f368e4265484e0e52507dfbffef970feac2caf859ba88689d6dd7bbf2e WatchSource:0}: Error finding container 160da2f368e4265484e0e52507dfbffef970feac2caf859ba88689d6dd7bbf2e: Status 404 returned error can't find the container with id 160da2f368e4265484e0e52507dfbffef970feac2caf859ba88689d6dd7bbf2e Oct 30 17:12:24.157478 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:24.157343 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-22vlj" event={"ID":"2939b3a2-1950-44c9-802c-a53861aa7aa8","Type":"ContainerStarted","Data":"3ba04c06cf73d36bd12eb8870289633020741c397ca5f12b7bc6de365ade792a"} Oct 30 17:12:24.160383 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:24.160343 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 17:12:24.160524 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:24.160435 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" event={"ID":"1eab122e-07da-4307-b7f2-914cbe2b4237","Type":"ContainerStarted","Data":"6d321327d49a5903651e47900987e40ca4d3b84c77198eb6ea63ce9b4e2e5362"} Oct 30 17:12:24.162603 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:24.162556 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-kvpw6" event={"ID":"c2d3293d-1890-4fa2-b790-9f84622c9c8b","Type":"ContainerStarted","Data":"3e090e3fe9da270184bc8d6fdac9174f2dd56dc6b4e29fdd4653ff273fa6d479"} Oct 30 17:12:24.164940 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:24.164917 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" event={"ID":"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce","Type":"ContainerStarted","Data":"d2d17cf7af9a1aabade19d16f36f5bf7411cbbaceadf0763c677de2b9e050ce5"} Oct 30 17:12:24.165027 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:24.164957 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" event={"ID":"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce","Type":"ContainerStarted","Data":"1db9d57d93616d600cb967e4161c6118f8991004bdfda9c4b38a8f706776ba0d"} Oct 30 17:12:24.166949 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:24.166921 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t9hdn" event={"ID":"745f4f0f-74f7-4fff-9384-c8f45f601e33","Type":"ContainerStarted","Data":"0eeec9e437aac1e0b93b86105f73e07c2bb0d50273596bfcbc92bd6936361b0b"} Oct 30 17:12:24.171898 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:24.170906 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77579944f4-hctng" event={"ID":"fd74ba94-86d0-4707-94e5-6822c4ab5751","Type":"ContainerStarted","Data":"2241b433a010f7fa0c5a58ba334083de72fe0db0ac3af0e4a65b6aaec9b26672"} Oct 30 17:12:24.171898 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:24.170943 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77579944f4-hctng" event={"ID":"fd74ba94-86d0-4707-94e5-6822c4ab5751","Type":"ContainerStarted","Data":"160da2f368e4265484e0e52507dfbffef970feac2caf859ba88689d6dd7bbf2e"} Oct 30 17:12:24.721228 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:24.721196 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:24.842124 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:24.842062 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc71466d-eac0-460b-af50-3204f07b3f35-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6jbgg\" (UID: \"cc71466d-eac0-460b-af50-3204f07b3f35\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jbgg" Oct 30 17:12:24.842308 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:24.842119 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6084e9b4-6b79-46ab-91cf-8d292d1f4591-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:24.842308 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:24.842198 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6178534d-9f75-49c7-a977-de65737cbcea-metrics-certs\") pod \"network-metrics-daemon-87jjv\" (UID: \"6178534d-9f75-49c7-a977-de65737cbcea\") " pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:12:24.842308 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:24.842217 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-metrics-certs\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:24.842472 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:24.842333 4965 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 17:12:24.842472 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:24.842378 4965 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-metrics-certs podName:924ac066-ff98-4320-9eb2-268f0a39ca33 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:26.842366173 +0000 UTC m=+4.894466095 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-metrics-certs") pod "router-default-ddb54c4f7-54tw2" (UID: "924ac066-ff98-4320-9eb2-268f0a39ca33") : secret "router-metrics-certs-default" not found Oct 30 17:12:24.843040 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:24.843013 4965 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:12:24.843154 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:24.843082 4965 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6084e9b4-6b79-46ab-91cf-8d292d1f4591-insights-runtime-extractor-tls podName:6084e9b4-6b79-46ab-91cf-8d292d1f4591 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:26.843065264 +0000 UTC m=+4.895165187 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6084e9b4-6b79-46ab-91cf-8d292d1f4591-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbqhc" (UID: "6084e9b4-6b79-46ab-91cf-8d292d1f4591") : secret "insights-runtime-extractor-tls" not found Oct 30 17:12:24.843154 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:24.843139 4965 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 17:12:24.843293 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:24.843172 4965 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc71466d-eac0-460b-af50-3204f07b3f35-cluster-monitoring-operator-tls podName:cc71466d-eac0-460b-af50-3204f07b3f35 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:26.843162385 +0000 UTC m=+4.895262335 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc71466d-eac0-460b-af50-3204f07b3f35-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6jbgg" (UID: "cc71466d-eac0-460b-af50-3204f07b3f35") : secret "cluster-monitoring-operator-tls" not found Oct 30 17:12:24.845796 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:24.845769 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6178534d-9f75-49c7-a977-de65737cbcea-metrics-certs\") pod \"network-metrics-daemon-87jjv\" (UID: \"6178534d-9f75-49c7-a977-de65737cbcea\") " pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:12:24.899478 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:24.899446 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87jjv" Oct 30 17:12:25.176449 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:25.175851 4965 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:12:25.176449 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:25.175878 4965 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:12:25.214699 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:25.214671 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-87jjv"] Oct 30 17:12:25.218892 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:25.218814 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6178534d_9f75_49c7_a977_de65737cbcea.slice/crio-26597936d3de25c334d309143bd4082cc7a9c1bec568a79816ea6d9c9c8d8834 WatchSource:0}: Error finding container 26597936d3de25c334d309143bd4082cc7a9c1bec568a79816ea6d9c9c8d8834: Status 404 returned error can't find the container with id 26597936d3de25c334d309143bd4082cc7a9c1bec568a79816ea6d9c9c8d8834 Oct 30 17:12:26.181803 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:26.181771 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-22vlj" event={"ID":"2939b3a2-1950-44c9-802c-a53861aa7aa8","Type":"ContainerStarted","Data":"6ffac8a0ec771d3b3ecf4dd916fb305554bf4a0fdba4fd3acb232c378304d5af"} Oct 30 17:12:26.182244 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:26.181896 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-22vlj" Oct 30 17:12:26.182244 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:26.181926 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-22vlj" event={"ID":"2939b3a2-1950-44c9-802c-a53861aa7aa8","Type":"ContainerStarted","Data":"ac2a9a7730d923de6830aa8de540b7d1b5ede3042621e291e514d9c3a7e7540a"} Oct 30 17:12:26.184364 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:26.184325 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-kvpw6" event={"ID":"c2d3293d-1890-4fa2-b790-9f84622c9c8b","Type":"ContainerStarted","Data":"ea52048f217e161f09c2a082d0afca0c5fd0bb3880ff4fcbea6541fad5f2ebd0"} Oct 30 17:12:26.185802 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:26.185779 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-87jjv" event={"ID":"6178534d-9f75-49c7-a977-de65737cbcea","Type":"ContainerStarted","Data":"26597936d3de25c334d309143bd4082cc7a9c1bec568a79816ea6d9c9c8d8834"} Oct 30 17:12:26.188185 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:26.187715 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t9hdn" event={"ID":"745f4f0f-74f7-4fff-9384-c8f45f601e33","Type":"ContainerStarted","Data":"2ff237d4a90c46a091619fe072fb904b8eba107afd6cddc41242b288a3b78424"} Oct 30 17:12:26.188185 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:26.187758 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t9hdn" event={"ID":"745f4f0f-74f7-4fff-9384-c8f45f601e33","Type":"ContainerStarted","Data":"abcc82b20aa0766e7ba31b628b62f0bc85124e3492afbc6e7510c67a8951212c"} Oct 30 17:12:26.239254 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:26.239230 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:26.239433 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:26.239420 4965 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:12:26.787075 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:26.787035 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" Oct 30 17:12:26.787241 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:26.787192 4965 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:12:26.791884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:26.791846 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-mf9rd" Oct 30 17:12:26.864606 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:26.864566 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc71466d-eac0-460b-af50-3204f07b3f35-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6jbgg\" (UID: \"cc71466d-eac0-460b-af50-3204f07b3f35\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jbgg" Oct 30 17:12:26.864606 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:26.864606 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6084e9b4-6b79-46ab-91cf-8d292d1f4591-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:26.864853 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:26.864674 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-metrics-certs\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:26.864853 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:26.864711 4965 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 17:12:26.864853 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:26.864769 4965 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:12:26.864853 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:26.864786 4965 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc71466d-eac0-460b-af50-3204f07b3f35-cluster-monitoring-operator-tls podName:cc71466d-eac0-460b-af50-3204f07b3f35 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:30.864764737 +0000 UTC m=+8.916864666 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc71466d-eac0-460b-af50-3204f07b3f35-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6jbgg" (UID: "cc71466d-eac0-460b-af50-3204f07b3f35") : secret "cluster-monitoring-operator-tls" not found Oct 30 17:12:26.864853 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:26.864823 4965 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6084e9b4-6b79-46ab-91cf-8d292d1f4591-insights-runtime-extractor-tls podName:6084e9b4-6b79-46ab-91cf-8d292d1f4591 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:30.864808388 +0000 UTC m=+8.916908311 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6084e9b4-6b79-46ab-91cf-8d292d1f4591-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbqhc" (UID: "6084e9b4-6b79-46ab-91cf-8d292d1f4591") : secret "insights-runtime-extractor-tls" not found Oct 30 17:12:26.865124 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:26.864906 4965 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 17:12:26.865124 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:26.864946 4965 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-metrics-certs podName:924ac066-ff98-4320-9eb2-268f0a39ca33 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:30.86493417 +0000 UTC m=+8.917034115 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-metrics-certs") pod "router-default-ddb54c4f7-54tw2" (UID: "924ac066-ff98-4320-9eb2-268f0a39ca33") : secret "router-metrics-certs-default" not found Oct 30 17:12:27.196468 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:27.196418 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-87jjv" event={"ID":"6178534d-9f75-49c7-a977-de65737cbcea","Type":"ContainerStarted","Data":"0cde8ae3164bffa559fe151fa0d44c3168bb641d7f0e481a77779935226497c4"} Oct 30 17:12:27.197006 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:27.196523 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-87jjv" event={"ID":"6178534d-9f75-49c7-a977-de65737cbcea","Type":"ContainerStarted","Data":"2b661dc08baf2af111477915326caeb06791ab20d5938c1aa1d47a8867d0f68e"} Oct 30 17:12:27.197934 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:27.197910 4965 generic.go:356] "Generic (PLEG): container finished" podID="16cd937e-4c6d-45d0-a9de-f77dc836d082" containerID="d0ac680e4381e32ffe18de9c6cd9a7238a418be517a8c6fa8c6dd0d288673733" exitCode=0 Oct 30 17:12:27.198023 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:27.197998 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4w8fl" event={"ID":"16cd937e-4c6d-45d0-a9de-f77dc836d082","Type":"ContainerDied","Data":"d0ac680e4381e32ffe18de9c6cd9a7238a418be517a8c6fa8c6dd0d288673733"} Oct 30 17:12:27.198567 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:27.198549 4965 scope.go:122] "RemoveContainer" containerID="d0ac680e4381e32ffe18de9c6cd9a7238a418be517a8c6fa8c6dd0d288673733" Oct 30 17:12:27.387356 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:27.387330 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w54kt" Oct 30 17:12:28.203438 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:28.203403 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4w8fl" event={"ID":"16cd937e-4c6d-45d0-a9de-f77dc836d082","Type":"ContainerStarted","Data":"c654105b733943dee5c08c0eabc25079203957327c1481ce32a4177931ed4bea"} Oct 30 17:12:28.549891 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:28.549798 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:30.900376 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:30.900334 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-metrics-certs\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:30.900851 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:30.900406 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc71466d-eac0-460b-af50-3204f07b3f35-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6jbgg\" (UID: \"cc71466d-eac0-460b-af50-3204f07b3f35\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jbgg" Oct 30 17:12:30.900851 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:30.900427 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6084e9b4-6b79-46ab-91cf-8d292d1f4591-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:30.902713 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:30.902679 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6084e9b4-6b79-46ab-91cf-8d292d1f4591-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbqhc\" (UID: \"6084e9b4-6b79-46ab-91cf-8d292d1f4591\") " pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:30.902838 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:30.902818 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/924ac066-ff98-4320-9eb2-268f0a39ca33-metrics-certs\") pod \"router-default-ddb54c4f7-54tw2\" (UID: \"924ac066-ff98-4320-9eb2-268f0a39ca33\") " pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:30.902838 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:30.902830 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc71466d-eac0-460b-af50-3204f07b3f35-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6jbgg\" (UID: \"cc71466d-eac0-460b-af50-3204f07b3f35\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jbgg" Oct 30 17:12:31.174066 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:31.173986 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t2pqm" Oct 30 17:12:31.189100 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:31.189065 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jbgg" Oct 30 17:12:31.189687 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:31.189666 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:31.202386 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:31.202212 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tbqhc" Oct 30 17:12:31.522980 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:31.522876 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-6jbgg"] Oct 30 17:12:31.526099 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:31.526070 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc71466d_eac0_460b_af50_3204f07b3f35.slice/crio-5a89ab0f153eada6212db9b3b8e06718f6e49d8197ed9babb1be697c59d9196e WatchSource:0}: Error finding container 5a89ab0f153eada6212db9b3b8e06718f6e49d8197ed9babb1be697c59d9196e: Status 404 returned error can't find the container with id 5a89ab0f153eada6212db9b3b8e06718f6e49d8197ed9babb1be697c59d9196e Oct 30 17:12:31.530989 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:31.530702 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924ac066_ff98_4320_9eb2_268f0a39ca33.slice/crio-419a1205c029f581b58e5b0b7ee0ffec1de22ecf2679242961ecf40065e371b4 WatchSource:0}: Error finding container 419a1205c029f581b58e5b0b7ee0ffec1de22ecf2679242961ecf40065e371b4: Status 404 returned error can't find the container with id 419a1205c029f581b58e5b0b7ee0ffec1de22ecf2679242961ecf40065e371b4 Oct 30 17:12:31.535798 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:31.535667 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-ddb54c4f7-54tw2"] Oct 30 17:12:31.536457 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:31.536435 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tbqhc"] Oct 30 17:12:31.541516 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:31.541492 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6084e9b4_6b79_46ab_91cf_8d292d1f4591.slice/crio-f9887541a8d3c03c91576bf925306e0505a241557bd038178bc6777a0bb859cd WatchSource:0}: Error finding container f9887541a8d3c03c91576bf925306e0505a241557bd038178bc6777a0bb859cd: Status 404 returned error can't find the container with id f9887541a8d3c03c91576bf925306e0505a241557bd038178bc6777a0bb859cd Oct 30 17:12:31.939474 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:31.939438 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:32.223235 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:32.221913 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-ddb54c4f7-54tw2" event={"ID":"924ac066-ff98-4320-9eb2-268f0a39ca33","Type":"ContainerStarted","Data":"af9a4bb4cb354f1609a8237bc007795791172b01b682bd9fbd8339ad7c668713"} Oct 30 17:12:32.223235 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:32.221964 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-ddb54c4f7-54tw2" event={"ID":"924ac066-ff98-4320-9eb2-268f0a39ca33","Type":"ContainerStarted","Data":"419a1205c029f581b58e5b0b7ee0ffec1de22ecf2679242961ecf40065e371b4"} Oct 30 17:12:32.224663 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:32.224634 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jbgg" event={"ID":"cc71466d-eac0-460b-af50-3204f07b3f35","Type":"ContainerStarted","Data":"5a89ab0f153eada6212db9b3b8e06718f6e49d8197ed9babb1be697c59d9196e"} Oct 30 17:12:32.226632 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:32.226597 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tbqhc" event={"ID":"6084e9b4-6b79-46ab-91cf-8d292d1f4591","Type":"ContainerStarted","Data":"e7fa2b8c1d56715f27a42bc1ebe72c19766bc402e53bcb403d536e2fb47583a3"} Oct 30 17:12:32.226804 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:32.226671 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tbqhc" event={"ID":"6084e9b4-6b79-46ab-91cf-8d292d1f4591","Type":"ContainerStarted","Data":"ede81034dee832e5e7862a460e61ba54d0e08b90b101b24210269a62a677fa63"} Oct 30 17:12:32.226804 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:32.226706 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tbqhc" event={"ID":"6084e9b4-6b79-46ab-91cf-8d292d1f4591","Type":"ContainerStarted","Data":"f9887541a8d3c03c91576bf925306e0505a241557bd038178bc6777a0bb859cd"} Oct 30 17:12:33.190824 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:33.190799 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:33.193558 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:33.193538 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:33.233233 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:33.233207 4965 generic.go:356] "Generic (PLEG): container finished" podID="16cd937e-4c6d-45d0-a9de-f77dc836d082" containerID="c654105b733943dee5c08c0eabc25079203957327c1481ce32a4177931ed4bea" exitCode=0 Oct 30 17:12:33.233346 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:33.233289 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4w8fl" event={"ID":"16cd937e-4c6d-45d0-a9de-f77dc836d082","Type":"ContainerDied","Data":"c654105b733943dee5c08c0eabc25079203957327c1481ce32a4177931ed4bea"} Oct 30 17:12:33.233392 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:33.233361 4965 scope.go:122] "RemoveContainer" containerID="d0ac680e4381e32ffe18de9c6cd9a7238a418be517a8c6fa8c6dd0d288673733" Oct 30 17:12:33.233890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:33.233738 4965 scope.go:122] "RemoveContainer" containerID="c654105b733943dee5c08c0eabc25079203957327c1481ce32a4177931ed4bea" Oct 30 17:12:33.233991 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:33.233962 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"insights-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=insights-operator pod=insights-operator-7c4986d449-4w8fl_openshift-insights(16cd937e-4c6d-45d0-a9de-f77dc836d082)\"" pod="openshift-insights/insights-operator-7c4986d449-4w8fl" podUID="16cd937e-4c6d-45d0-a9de-f77dc836d082" Oct 30 17:12:34.241939 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:34.241896 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tbqhc" event={"ID":"6084e9b4-6b79-46ab-91cf-8d292d1f4591","Type":"ContainerStarted","Data":"87db34e6d91117d53be71b273428ee50821281a9180566b49bf709b9b7d9de44"} Oct 30 17:12:34.244790 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:34.244772 4965 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:12:34.244935 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:34.244792 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6jbgg" event={"ID":"cc71466d-eac0-460b-af50-3204f07b3f35","Type":"ContainerStarted","Data":"64e157efa16d313ddd15fd86d1459e5bd77f6c923ef660c104b792d2411012dd"} Oct 30 17:12:35.994905 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:12:35.994849 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:12:36.176325 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.176285 4965 patch_prober.go:29] interesting pod/image-registry-585b9dc6c5-dblkr container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 17:12:36.176479 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.176338 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" podUID="b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:12:36.200362 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.200333 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-22vlj" Oct 30 17:12:36.647458 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.647426 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-fngsc-t5d6p"] Oct 30 17:12:36.651387 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.651366 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fngsc-t5d6p" Oct 30 17:12:36.662478 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.662452 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-j6pg9"] Oct 30 17:12:36.666025 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.666005 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-s7vbz"] Oct 30 17:12:36.666120 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.666102 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-j6pg9" Oct 30 17:12:36.669561 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.669543 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s7vbz" Oct 30 17:12:36.675333 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.675309 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-fngsc-t5d6p"] Oct 30 17:12:36.684316 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.684274 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-m65sr"] Oct 30 17:12:36.687434 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.687414 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-585b9dc6c5-dblkr"] Oct 30 17:12:36.687518 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.687502 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m65sr" Oct 30 17:12:36.693651 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.693598 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 17:12:36.695171 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.695151 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-j6pg9"] Oct 30 17:12:36.697244 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.695350 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 17:12:36.697244 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.695438 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 17:12:36.697244 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.695551 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-hcpj4\"" Oct 30 17:12:36.697244 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.695642 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-krhw6\"" Oct 30 17:12:36.697244 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.696714 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-s7vbz"] Oct 30 17:12:36.697734 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.697712 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 17:12:36.723659 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.723623 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-m65sr"] Oct 30 17:12:36.730803 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.730780 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65f579c9f9-fhj5r"] Oct 30 17:12:36.733992 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.733973 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65f579c9f9-fhj5r" Oct 30 17:12:36.751414 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.751382 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65986964-0b39-48e4-9163-705ed74afd18-service-ca-bundle\") pod \"periodic-gathering-fngsc-t5d6p\" (UID: \"65986964-0b39-48e4-9163-705ed74afd18\") " pod="openshift-insights/periodic-gathering-fngsc-t5d6p" Oct 30 17:12:36.751551 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.751439 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfznx\" (UniqueName: \"kubernetes.io/projected/65986964-0b39-48e4-9163-705ed74afd18-kube-api-access-tfznx\") pod \"periodic-gathering-fngsc-t5d6p\" (UID: \"65986964-0b39-48e4-9163-705ed74afd18\") " pod="openshift-insights/periodic-gathering-fngsc-t5d6p" Oct 30 17:12:36.751551 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.751499 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65986964-0b39-48e4-9163-705ed74afd18-trusted-ca-bundle\") pod \"periodic-gathering-fngsc-t5d6p\" (UID: \"65986964-0b39-48e4-9163-705ed74afd18\") " pod="openshift-insights/periodic-gathering-fngsc-t5d6p" Oct 30 17:12:36.751625 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.751601 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/65986964-0b39-48e4-9163-705ed74afd18-archives-path\") pod \"periodic-gathering-fngsc-t5d6p\" (UID: \"65986964-0b39-48e4-9163-705ed74afd18\") " pod="openshift-insights/periodic-gathering-fngsc-t5d6p" Oct 30 17:12:36.759027 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.759000 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65f579c9f9-fhj5r"] Oct 30 17:12:36.852319 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.852289 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfznx\" (UniqueName: \"kubernetes.io/projected/65986964-0b39-48e4-9163-705ed74afd18-kube-api-access-tfznx\") pod \"periodic-gathering-fngsc-t5d6p\" (UID: \"65986964-0b39-48e4-9163-705ed74afd18\") " pod="openshift-insights/periodic-gathering-fngsc-t5d6p" Oct 30 17:12:36.852319 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.852323 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/146cf698-06cf-47f6-a8fc-374eff2141be-registry-tls\") pod \"image-registry-65f579c9f9-fhj5r\" (UID: \"146cf698-06cf-47f6-a8fc-374eff2141be\") " pod="openshift-image-registry/image-registry-65f579c9f9-fhj5r" Oct 30 17:12:36.852522 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.852343 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65986964-0b39-48e4-9163-705ed74afd18-trusted-ca-bundle\") pod \"periodic-gathering-fngsc-t5d6p\" (UID: \"65986964-0b39-48e4-9163-705ed74afd18\") " pod="openshift-insights/periodic-gathering-fngsc-t5d6p" Oct 30 17:12:36.852522 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.852375 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftmgc\" (UniqueName: \"kubernetes.io/projected/99c91f4b-d893-45d0-82a0-abbb240a49e7-kube-api-access-ftmgc\") pod \"downloads-6dd789c8b7-j6pg9\" (UID: \"99c91f4b-d893-45d0-82a0-abbb240a49e7\") " pod="openshift-console/downloads-6dd789c8b7-j6pg9" Oct 30 17:12:36.852522 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.852464 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/146cf698-06cf-47f6-a8fc-374eff2141be-trusted-ca\") pod \"image-registry-65f579c9f9-fhj5r\" (UID: \"146cf698-06cf-47f6-a8fc-374eff2141be\") " pod="openshift-image-registry/image-registry-65f579c9f9-fhj5r" Oct 30 17:12:36.852522 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.852498 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/146cf698-06cf-47f6-a8fc-374eff2141be-installation-pull-secrets\") pod \"image-registry-65f579c9f9-fhj5r\" (UID: \"146cf698-06cf-47f6-a8fc-374eff2141be\") " pod="openshift-image-registry/image-registry-65f579c9f9-fhj5r" Oct 30 17:12:36.852712 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.852527 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhbpt\" (UniqueName: \"kubernetes.io/projected/146cf698-06cf-47f6-a8fc-374eff2141be-kube-api-access-dhbpt\") pod \"image-registry-65f579c9f9-fhj5r\" (UID: \"146cf698-06cf-47f6-a8fc-374eff2141be\") " pod="openshift-image-registry/image-registry-65f579c9f9-fhj5r" Oct 30 17:12:36.852712 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.852574 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64hxf\" (UniqueName: \"kubernetes.io/projected/46515ca5-213a-4acc-8db2-680096edc147-kube-api-access-64hxf\") pod \"migrator-66df5d496b-m65sr\" (UID: \"46515ca5-213a-4acc-8db2-680096edc147\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m65sr" Oct 30 17:12:36.852712 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.852630 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/146cf698-06cf-47f6-a8fc-374eff2141be-registry-certificates\") pod \"image-registry-65f579c9f9-fhj5r\" (UID: \"146cf698-06cf-47f6-a8fc-374eff2141be\") " pod="openshift-image-registry/image-registry-65f579c9f9-fhj5r" Oct 30 17:12:36.852712 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.852682 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9zg7\" (UniqueName: \"kubernetes.io/projected/0ea73d05-f4cd-45a4-b043-d0ec3391235e-kube-api-access-l9zg7\") pod \"migrator-66df5d496b-s7vbz\" (UID: \"0ea73d05-f4cd-45a4-b043-d0ec3391235e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s7vbz" Oct 30 17:12:36.852712 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.852710 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/65986964-0b39-48e4-9163-705ed74afd18-archives-path\") pod \"periodic-gathering-fngsc-t5d6p\" (UID: \"65986964-0b39-48e4-9163-705ed74afd18\") " pod="openshift-insights/periodic-gathering-fngsc-t5d6p" Oct 30 17:12:36.852934 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.852749 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/146cf698-06cf-47f6-a8fc-374eff2141be-bound-sa-token\") pod \"image-registry-65f579c9f9-fhj5r\" (UID: \"146cf698-06cf-47f6-a8fc-374eff2141be\") " pod="openshift-image-registry/image-registry-65f579c9f9-fhj5r" Oct 30 17:12:36.852934 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.852790 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99c91f4b-d893-45d0-82a0-abbb240a49e7-tmp\") pod \"downloads-6dd789c8b7-j6pg9\" (UID: \"99c91f4b-d893-45d0-82a0-abbb240a49e7\") " pod="openshift-console/downloads-6dd789c8b7-j6pg9" Oct 30 17:12:36.852934 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.852817 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65986964-0b39-48e4-9163-705ed74afd18-service-ca-bundle\") pod \"periodic-gathering-fngsc-t5d6p\" (UID: \"65986964-0b39-48e4-9163-705ed74afd18\") " pod="openshift-insights/periodic-gathering-fngsc-t5d6p" Oct 30 17:12:36.852934 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.852886 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/146cf698-06cf-47f6-a8fc-374eff2141be-image-registry-private-configuration\") pod \"image-registry-65f579c9f9-fhj5r\" (UID: \"146cf698-06cf-47f6-a8fc-374eff2141be\") " pod="openshift-image-registry/image-registry-65f579c9f9-fhj5r" Oct 30 17:12:36.852934 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.852917 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/146cf698-06cf-47f6-a8fc-374eff2141be-ca-trust-extracted\") pod \"image-registry-65f579c9f9-fhj5r\" (UID: \"146cf698-06cf-47f6-a8fc-374eff2141be\") " pod="openshift-image-registry/image-registry-65f579c9f9-fhj5r" Oct 30 17:12:36.853236 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.853082 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/65986964-0b39-48e4-9163-705ed74afd18-archives-path\") pod \"periodic-gathering-fngsc-t5d6p\" (UID: \"65986964-0b39-48e4-9163-705ed74afd18\") " pod="openshift-insights/periodic-gathering-fngsc-t5d6p" Oct 30 17:12:36.853327 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.853307 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65986964-0b39-48e4-9163-705ed74afd18-trusted-ca-bundle\") pod \"periodic-gathering-fngsc-t5d6p\" (UID: \"65986964-0b39-48e4-9163-705ed74afd18\") " pod="openshift-insights/periodic-gathering-fngsc-t5d6p" Oct 30 17:12:36.853490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.853462 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65986964-0b39-48e4-9163-705ed74afd18-service-ca-bundle\") pod \"periodic-gathering-fngsc-t5d6p\" (UID: \"65986964-0b39-48e4-9163-705ed74afd18\") " pod="openshift-insights/periodic-gathering-fngsc-t5d6p" Oct 30 17:12:36.875427 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.875403 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfznx\" (UniqueName: \"kubernetes.io/projected/65986964-0b39-48e4-9163-705ed74afd18-kube-api-access-tfznx\") pod \"periodic-gathering-fngsc-t5d6p\" (UID: \"65986964-0b39-48e4-9163-705ed74afd18\") " pod="openshift-insights/periodic-gathering-fngsc-t5d6p" Oct 30 17:12:36.953967 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.953904 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/146cf698-06cf-47f6-a8fc-374eff2141be-image-registry-private-configuration\") pod \"image-registry-65f579c9f9-fhj5r\" (UID: \"146cf698-06cf-47f6-a8fc-374eff2141be\") " pod="openshift-image-registry/image-registry-65f579c9f9-fhj5r" Oct 30 17:12:36.953967 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.953940 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/146cf698-06cf-47f6-a8fc-374eff2141be-ca-trust-extracted\") pod \"image-registry-65f579c9f9-fhj5r\" (UID: \"146cf698-06cf-47f6-a8fc-374eff2141be\") " pod="openshift-image-registry/image-registry-65f579c9f9-fhj5r" Oct 30 17:12:36.954145 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.953978 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/146cf698-06cf-47f6-a8fc-374eff2141be-registry-tls\") pod \"image-registry-65f579c9f9-fhj5r\" (UID: \"146cf698-06cf-47f6-a8fc-374eff2141be\") " pod="openshift-image-registry/image-registry-65f579c9f9-fhj5r" Oct 30 17:12:36.954145 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.954011 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftmgc\" (UniqueName: \"kubernetes.io/projected/99c91f4b-d893-45d0-82a0-abbb240a49e7-kube-api-access-ftmgc\") pod \"downloads-6dd789c8b7-j6pg9\" (UID: \"99c91f4b-d893-45d0-82a0-abbb240a49e7\") " pod="openshift-console/downloads-6dd789c8b7-j6pg9" Oct 30 17:12:36.954145 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.954037 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/146cf698-06cf-47f6-a8fc-374eff2141be-trusted-ca\") pod \"image-registry-65f579c9f9-fhj5r\" (UID: \"146cf698-06cf-47f6-a8fc-374eff2141be\") " pod="openshift-image-registry/image-registry-65f579c9f9-fhj5r" Oct 30 17:12:36.954145 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.954126 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/146cf698-06cf-47f6-a8fc-374eff2141be-installation-pull-secrets\") pod \"image-registry-65f579c9f9-fhj5r\" (UID: \"146cf698-06cf-47f6-a8fc-374eff2141be\") " pod="openshift-image-registry/image-registry-65f579c9f9-fhj5r" Oct 30 17:12:36.954367 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.954169 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhbpt\" (UniqueName: \"kubernetes.io/projected/146cf698-06cf-47f6-a8fc-374eff2141be-kube-api-access-dhbpt\") pod \"image-registry-65f579c9f9-fhj5r\" (UID: \"146cf698-06cf-47f6-a8fc-374eff2141be\") " pod="openshift-image-registry/image-registry-65f579c9f9-fhj5r" Oct 30 17:12:36.954367 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.954205 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64hxf\" (UniqueName: \"kubernetes.io/projected/46515ca5-213a-4acc-8db2-680096edc147-kube-api-access-64hxf\") pod \"migrator-66df5d496b-m65sr\" (UID: \"46515ca5-213a-4acc-8db2-680096edc147\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m65sr" Oct 30 17:12:36.954367 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.954229 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/146cf698-06cf-47f6-a8fc-374eff2141be-registry-certificates\") pod \"image-registry-65f579c9f9-fhj5r\" (UID: \"146cf698-06cf-47f6-a8fc-374eff2141be\") " pod="openshift-image-registry/image-registry-65f579c9f9-fhj5r" Oct 30 17:12:36.954367 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.954267 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9zg7\" (UniqueName: \"kubernetes.io/projected/0ea73d05-f4cd-45a4-b043-d0ec3391235e-kube-api-access-l9zg7\") pod \"migrator-66df5d496b-s7vbz\" (UID: \"0ea73d05-f4cd-45a4-b043-d0ec3391235e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s7vbz" Oct 30 17:12:36.954367 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.954326 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/146cf698-06cf-47f6-a8fc-374eff2141be-bound-sa-token\") pod \"image-registry-65f579c9f9-fhj5r\" (UID: \"146cf698-06cf-47f6-a8fc-374eff2141be\") " pod="openshift-image-registry/image-registry-65f579c9f9-fhj5r" Oct 30 17:12:36.954367 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.954356 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99c91f4b-d893-45d0-82a0-abbb240a49e7-tmp\") pod \"downloads-6dd789c8b7-j6pg9\" (UID: \"99c91f4b-d893-45d0-82a0-abbb240a49e7\") " pod="openshift-console/downloads-6dd789c8b7-j6pg9" Oct 30 17:12:36.954734 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.954542 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/146cf698-06cf-47f6-a8fc-374eff2141be-ca-trust-extracted\") pod \"image-registry-65f579c9f9-fhj5r\" (UID: \"146cf698-06cf-47f6-a8fc-374eff2141be\") " pod="openshift-image-registry/image-registry-65f579c9f9-fhj5r" Oct 30 17:12:36.954835 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.954807 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99c91f4b-d893-45d0-82a0-abbb240a49e7-tmp\") pod \"downloads-6dd789c8b7-j6pg9\" (UID: \"99c91f4b-d893-45d0-82a0-abbb240a49e7\") " pod="openshift-console/downloads-6dd789c8b7-j6pg9" Oct 30 17:12:36.955179 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.955153 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/146cf698-06cf-47f6-a8fc-374eff2141be-trusted-ca\") pod \"image-registry-65f579c9f9-fhj5r\" (UID: \"146cf698-06cf-47f6-a8fc-374eff2141be\") " pod="openshift-image-registry/image-registry-65f579c9f9-fhj5r" Oct 30 17:12:36.955299 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.955281 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/146cf698-06cf-47f6-a8fc-374eff2141be-registry-certificates\") pod \"image-registry-65f579c9f9-fhj5r\" (UID: \"146cf698-06cf-47f6-a8fc-374eff2141be\") " pod="openshift-image-registry/image-registry-65f579c9f9-fhj5r" Oct 30 17:12:36.956596 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.956570 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/146cf698-06cf-47f6-a8fc-374eff2141be-registry-tls\") pod \"image-registry-65f579c9f9-fhj5r\" (UID: \"146cf698-06cf-47f6-a8fc-374eff2141be\") " pod="openshift-image-registry/image-registry-65f579c9f9-fhj5r" Oct 30 17:12:36.956719 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.956696 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/146cf698-06cf-47f6-a8fc-374eff2141be-image-registry-private-configuration\") pod \"image-registry-65f579c9f9-fhj5r\" (UID: \"146cf698-06cf-47f6-a8fc-374eff2141be\") " pod="openshift-image-registry/image-registry-65f579c9f9-fhj5r" Oct 30 17:12:36.956982 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.956963 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/146cf698-06cf-47f6-a8fc-374eff2141be-installation-pull-secrets\") pod \"image-registry-65f579c9f9-fhj5r\" (UID: \"146cf698-06cf-47f6-a8fc-374eff2141be\") " pod="openshift-image-registry/image-registry-65f579c9f9-fhj5r" Oct 30 17:12:36.963909 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.963848 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fngsc-t5d6p" Oct 30 17:12:36.980602 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.980552 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9zg7\" (UniqueName: \"kubernetes.io/projected/0ea73d05-f4cd-45a4-b043-d0ec3391235e-kube-api-access-l9zg7\") pod \"migrator-66df5d496b-s7vbz\" (UID: \"0ea73d05-f4cd-45a4-b043-d0ec3391235e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s7vbz" Oct 30 17:12:36.981170 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.981142 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhbpt\" (UniqueName: \"kubernetes.io/projected/146cf698-06cf-47f6-a8fc-374eff2141be-kube-api-access-dhbpt\") pod \"image-registry-65f579c9f9-fhj5r\" (UID: \"146cf698-06cf-47f6-a8fc-374eff2141be\") " pod="openshift-image-registry/image-registry-65f579c9f9-fhj5r" Oct 30 17:12:36.981312 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.981289 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftmgc\" (UniqueName: \"kubernetes.io/projected/99c91f4b-d893-45d0-82a0-abbb240a49e7-kube-api-access-ftmgc\") pod \"downloads-6dd789c8b7-j6pg9\" (UID: \"99c91f4b-d893-45d0-82a0-abbb240a49e7\") " pod="openshift-console/downloads-6dd789c8b7-j6pg9" Oct 30 17:12:36.981916 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.981856 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/146cf698-06cf-47f6-a8fc-374eff2141be-bound-sa-token\") pod \"image-registry-65f579c9f9-fhj5r\" (UID: \"146cf698-06cf-47f6-a8fc-374eff2141be\") " pod="openshift-image-registry/image-registry-65f579c9f9-fhj5r" Oct 30 17:12:36.982128 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.982094 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64hxf\" (UniqueName: \"kubernetes.io/projected/46515ca5-213a-4acc-8db2-680096edc147-kube-api-access-64hxf\") pod \"migrator-66df5d496b-m65sr\" (UID: \"46515ca5-213a-4acc-8db2-680096edc147\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m65sr" Oct 30 17:12:36.986781 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.986761 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-j6pg9" Oct 30 17:12:36.989408 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:36.989387 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s7vbz" Oct 30 17:12:37.002853 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:37.002825 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m65sr" Oct 30 17:12:37.046605 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:37.046578 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65f579c9f9-fhj5r" Oct 30 17:12:37.266929 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:37.266905 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-fngsc-t5d6p"] Oct 30 17:12:37.269272 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:37.269242 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65986964_0b39_48e4_9163_705ed74afd18.slice/crio-4ff35bbc71b9691038fdb167644c1d6d8496215f3c709f8dd8c7267de461984f WatchSource:0}: Error finding container 4ff35bbc71b9691038fdb167644c1d6d8496215f3c709f8dd8c7267de461984f: Status 404 returned error can't find the container with id 4ff35bbc71b9691038fdb167644c1d6d8496215f3c709f8dd8c7267de461984f Oct 30 17:12:37.312789 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:37.312762 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-s7vbz"] Oct 30 17:12:37.315770 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:37.315749 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ea73d05_f4cd_45a4_b043_d0ec3391235e.slice/crio-ed2f0ae28e3d5e64dbf768f3cb204c58987bff28a6c607387e96feb070b7d027 WatchSource:0}: Error finding container ed2f0ae28e3d5e64dbf768f3cb204c58987bff28a6c607387e96feb070b7d027: Status 404 returned error can't find the container with id ed2f0ae28e3d5e64dbf768f3cb204c58987bff28a6c607387e96feb070b7d027 Oct 30 17:12:37.326940 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:37.326919 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-j6pg9"] Oct 30 17:12:37.328558 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:37.328502 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c91f4b_d893_45d0_82a0_abbb240a49e7.slice/crio-bdb8b5fcc01476cfe7b032f0cb2daf359664f9ef7eaf0b12051bbd8c476f2e31 WatchSource:0}: Error finding container bdb8b5fcc01476cfe7b032f0cb2daf359664f9ef7eaf0b12051bbd8c476f2e31: Status 404 returned error can't find the container with id bdb8b5fcc01476cfe7b032f0cb2daf359664f9ef7eaf0b12051bbd8c476f2e31 Oct 30 17:12:37.331266 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:37.331190 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-m65sr"] Oct 30 17:12:37.333295 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:37.333271 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46515ca5_213a_4acc_8db2_680096edc147.slice/crio-2f299c7cb623241906824ec4c39f6a210280ff5d0fa9337ac20ad6f807e2afc4 WatchSource:0}: Error finding container 2f299c7cb623241906824ec4c39f6a210280ff5d0fa9337ac20ad6f807e2afc4: Status 404 returned error can't find the container with id 2f299c7cb623241906824ec4c39f6a210280ff5d0fa9337ac20ad6f807e2afc4 Oct 30 17:12:37.373822 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:37.370134 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65f579c9f9-fhj5r"] Oct 30 17:12:37.376983 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:37.376948 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146cf698_06cf_47f6_a8fc_374eff2141be.slice/crio-2116130a28b61526256eb20768132a6b1b88d6e2e2184d5205235cb97c86f959 WatchSource:0}: Error finding container 2116130a28b61526256eb20768132a6b1b88d6e2e2184d5205235cb97c86f959: Status 404 returned error can't find the container with id 2116130a28b61526256eb20768132a6b1b88d6e2e2184d5205235cb97c86f959 Oct 30 17:12:38.265761 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:38.265720 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65f579c9f9-fhj5r" event={"ID":"146cf698-06cf-47f6-a8fc-374eff2141be","Type":"ContainerStarted","Data":"97e8d4464320dac6d9ac5430fdc91307faef4394e80ce41d6da76f82cb52bf69"} Oct 30 17:12:38.266258 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:38.265781 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65f579c9f9-fhj5r" event={"ID":"146cf698-06cf-47f6-a8fc-374eff2141be","Type":"ContainerStarted","Data":"2116130a28b61526256eb20768132a6b1b88d6e2e2184d5205235cb97c86f959"} Oct 30 17:12:38.266258 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:38.265933 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65f579c9f9-fhj5r" Oct 30 17:12:38.267619 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:38.267589 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m65sr" event={"ID":"46515ca5-213a-4acc-8db2-680096edc147","Type":"ContainerStarted","Data":"2f299c7cb623241906824ec4c39f6a210280ff5d0fa9337ac20ad6f807e2afc4"} Oct 30 17:12:38.270856 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:38.270832 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fngsc-t5d6p" event={"ID":"65986964-0b39-48e4-9163-705ed74afd18","Type":"ContainerStarted","Data":"d181ee8d25585aa0eea312407fa89fcb3f8df5c90f00363a3c3715b499c62817"} Oct 30 17:12:38.270987 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:38.270894 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fngsc-t5d6p" event={"ID":"65986964-0b39-48e4-9163-705ed74afd18","Type":"ContainerStarted","Data":"4ff35bbc71b9691038fdb167644c1d6d8496215f3c709f8dd8c7267de461984f"} Oct 30 17:12:38.272454 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:38.272357 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s7vbz" event={"ID":"0ea73d05-f4cd-45a4-b043-d0ec3391235e","Type":"ContainerStarted","Data":"ed2f0ae28e3d5e64dbf768f3cb204c58987bff28a6c607387e96feb070b7d027"} Oct 30 17:12:38.273737 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:38.273702 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-j6pg9" event={"ID":"99c91f4b-d893-45d0-82a0-abbb240a49e7","Type":"ContainerStarted","Data":"bdb8b5fcc01476cfe7b032f0cb2daf359664f9ef7eaf0b12051bbd8c476f2e31"} Oct 30 17:12:38.338350 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:38.338100 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-65f579c9f9-fhj5r" podStartSLOduration=2.338078336 podStartE2EDuration="2.338078336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:38.323989423 +0000 UTC m=+16.376089374" watchObservedRunningTime="2026-10-30 17:12:38.338078336 +0000 UTC m=+16.390178263" Oct 30 17:12:38.735343 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:38.735314 4965 patch_prober.go:29] interesting pod/image-registry-77579944f4-hctng container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 17:12:38.735439 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:38.735362 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-77579944f4-hctng" podUID="fd74ba94-86d0-4707-94e5-6822c4ab5751" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:12:39.281201 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:39.280977 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m65sr" event={"ID":"46515ca5-213a-4acc-8db2-680096edc147","Type":"ContainerStarted","Data":"b7f5ac1a4b4e1ff7c383ee4a7de176577962f269e444dd99fddf24151d3c32d4"} Oct 30 17:12:39.282876 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:39.282832 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s7vbz" event={"ID":"0ea73d05-f4cd-45a4-b043-d0ec3391235e","Type":"ContainerStarted","Data":"26bb0ff5f686bcf52903ba6ab479b519e5afaf08bf0017fa717dca0bb86799f9"} Oct 30 17:12:39.321775 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:39.321732 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-fngsc-t5d6p" podStartSLOduration=3.321717411 podStartE2EDuration="3.321717411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:38.36593808 +0000 UTC m=+16.418038025" watchObservedRunningTime="2026-10-30 17:12:39.321717411 +0000 UTC m=+17.373817338" Oct 30 17:12:39.322109 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:39.322074 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m65sr" podStartSLOduration=1.988970486 podStartE2EDuration="3.322067414s" totalImagesPullingTime="1.333096928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:39.321317127 +0000 UTC m=+17.373417070" watchObservedRunningTime="2026-10-30 17:12:39.322067414 +0000 UTC m=+17.374167340" Oct 30 17:12:39.362574 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:39.362526 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s7vbz" podStartSLOduration=2.016048739 podStartE2EDuration="3.362513658s" totalImagesPullingTime="1.346464919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:39.36217056 +0000 UTC m=+17.414270516" watchObservedRunningTime="2026-10-30 17:12:39.362513658 +0000 UTC m=+17.414613589" Oct 30 17:12:39.502106 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:39.502076 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:39.502289 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:39.502214 4965 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:12:39.503613 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:39.503587 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-ddb54c4f7-54tw2" Oct 30 17:12:39.997550 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:39.997480 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-22vlj_2939b3a2-1950-44c9-802c-a53861aa7aa8/dns/0.log" Oct 30 17:12:40.131809 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:40.131777 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-22vlj_2939b3a2-1950-44c9-802c-a53861aa7aa8/kube-rbac-proxy/0.log" Oct 30 17:12:40.530852 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:40.530820 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zgb2b_467187ee-7039-4bb4-adbe-38161ee7c336/dns-node-resolver/0.log" Oct 30 17:12:40.931411 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:40.931375 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-585b9dc6c5-dblkr_b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce/registry/0.log" Oct 30 17:12:41.135074 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.135047 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-65f579c9f9-fhj5r_146cf698-06cf-47f6-a8fc-374eff2141be/registry/0.log" Oct 30 17:12:41.145512 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.145481 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c6cc76744-s2pww"] Oct 30 17:12:41.149163 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.149136 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6cc76744-s2pww" Oct 30 17:12:41.168528 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.168498 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c6cc76744-s2pww"] Oct 30 17:12:41.178807 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.178744 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 17:12:41.179025 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.179002 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 17:12:41.179025 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.179001 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 17:12:41.179162 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.179127 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 17:12:41.267009 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.266939 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 17:12:41.267009 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.266997 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vqnr7\"" Oct 30 17:12:41.293563 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.293527 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/072395f0-b9b7-438a-83d6-9f92feb714ca-service-ca\") pod \"console-7c6cc76744-s2pww\" (UID: \"072395f0-b9b7-438a-83d6-9f92feb714ca\") " pod="openshift-console/console-7c6cc76744-s2pww" Oct 30 17:12:41.293694 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.293573 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/072395f0-b9b7-438a-83d6-9f92feb714ca-console-serving-cert\") pod \"console-7c6cc76744-s2pww\" (UID: \"072395f0-b9b7-438a-83d6-9f92feb714ca\") " pod="openshift-console/console-7c6cc76744-s2pww" Oct 30 17:12:41.293694 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.293649 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/072395f0-b9b7-438a-83d6-9f92feb714ca-console-oauth-config\") pod \"console-7c6cc76744-s2pww\" (UID: \"072395f0-b9b7-438a-83d6-9f92feb714ca\") " pod="openshift-console/console-7c6cc76744-s2pww" Oct 30 17:12:41.293807 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.293709 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/072395f0-b9b7-438a-83d6-9f92feb714ca-console-config\") pod \"console-7c6cc76744-s2pww\" (UID: \"072395f0-b9b7-438a-83d6-9f92feb714ca\") " pod="openshift-console/console-7c6cc76744-s2pww" Oct 30 17:12:41.293807 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.293767 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/072395f0-b9b7-438a-83d6-9f92feb714ca-oauth-serving-cert\") pod \"console-7c6cc76744-s2pww\" (UID: \"072395f0-b9b7-438a-83d6-9f92feb714ca\") " pod="openshift-console/console-7c6cc76744-s2pww" Oct 30 17:12:41.293807 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.293789 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdc9z\" (UniqueName: \"kubernetes.io/projected/072395f0-b9b7-438a-83d6-9f92feb714ca-kube-api-access-bdc9z\") pod \"console-7c6cc76744-s2pww\" (UID: \"072395f0-b9b7-438a-83d6-9f92feb714ca\") " pod="openshift-console/console-7c6cc76744-s2pww" Oct 30 17:12:41.293986 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.293813 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/072395f0-b9b7-438a-83d6-9f92feb714ca-tmp\") pod \"console-7c6cc76744-s2pww\" (UID: \"072395f0-b9b7-438a-83d6-9f92feb714ca\") " pod="openshift-console/console-7c6cc76744-s2pww" Oct 30 17:12:41.331550 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.331526 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-77579944f4-hctng_fd74ba94-86d0-4707-94e5-6822c4ab5751/registry/0.log" Oct 30 17:12:41.394456 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.394413 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/072395f0-b9b7-438a-83d6-9f92feb714ca-oauth-serving-cert\") pod \"console-7c6cc76744-s2pww\" (UID: \"072395f0-b9b7-438a-83d6-9f92feb714ca\") " pod="openshift-console/console-7c6cc76744-s2pww" Oct 30 17:12:41.394456 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.394454 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdc9z\" (UniqueName: \"kubernetes.io/projected/072395f0-b9b7-438a-83d6-9f92feb714ca-kube-api-access-bdc9z\") pod \"console-7c6cc76744-s2pww\" (UID: \"072395f0-b9b7-438a-83d6-9f92feb714ca\") " pod="openshift-console/console-7c6cc76744-s2pww" Oct 30 17:12:41.394682 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.394547 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/072395f0-b9b7-438a-83d6-9f92feb714ca-tmp\") pod \"console-7c6cc76744-s2pww\" (UID: \"072395f0-b9b7-438a-83d6-9f92feb714ca\") " pod="openshift-console/console-7c6cc76744-s2pww" Oct 30 17:12:41.394682 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.394595 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/072395f0-b9b7-438a-83d6-9f92feb714ca-service-ca\") pod \"console-7c6cc76744-s2pww\" (UID: \"072395f0-b9b7-438a-83d6-9f92feb714ca\") " pod="openshift-console/console-7c6cc76744-s2pww" Oct 30 17:12:41.394682 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.394625 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/072395f0-b9b7-438a-83d6-9f92feb714ca-console-serving-cert\") pod \"console-7c6cc76744-s2pww\" (UID: \"072395f0-b9b7-438a-83d6-9f92feb714ca\") " pod="openshift-console/console-7c6cc76744-s2pww" Oct 30 17:12:41.394682 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.394652 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/072395f0-b9b7-438a-83d6-9f92feb714ca-console-oauth-config\") pod \"console-7c6cc76744-s2pww\" (UID: \"072395f0-b9b7-438a-83d6-9f92feb714ca\") " pod="openshift-console/console-7c6cc76744-s2pww" Oct 30 17:12:41.394889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.394692 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/072395f0-b9b7-438a-83d6-9f92feb714ca-console-config\") pod \"console-7c6cc76744-s2pww\" (UID: \"072395f0-b9b7-438a-83d6-9f92feb714ca\") " pod="openshift-console/console-7c6cc76744-s2pww" Oct 30 17:12:41.395250 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.395222 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/072395f0-b9b7-438a-83d6-9f92feb714ca-oauth-serving-cert\") pod \"console-7c6cc76744-s2pww\" (UID: \"072395f0-b9b7-438a-83d6-9f92feb714ca\") " pod="openshift-console/console-7c6cc76744-s2pww" Oct 30 17:12:41.395386 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.395253 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/072395f0-b9b7-438a-83d6-9f92feb714ca-console-config\") pod \"console-7c6cc76744-s2pww\" (UID: \"072395f0-b9b7-438a-83d6-9f92feb714ca\") " pod="openshift-console/console-7c6cc76744-s2pww" Oct 30 17:12:41.395386 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.395289 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/072395f0-b9b7-438a-83d6-9f92feb714ca-tmp\") pod \"console-7c6cc76744-s2pww\" (UID: \"072395f0-b9b7-438a-83d6-9f92feb714ca\") " pod="openshift-console/console-7c6cc76744-s2pww" Oct 30 17:12:41.395510 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.395485 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/072395f0-b9b7-438a-83d6-9f92feb714ca-service-ca\") pod \"console-7c6cc76744-s2pww\" (UID: \"072395f0-b9b7-438a-83d6-9f92feb714ca\") " pod="openshift-console/console-7c6cc76744-s2pww" Oct 30 17:12:41.398066 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.398041 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/072395f0-b9b7-438a-83d6-9f92feb714ca-console-oauth-config\") pod \"console-7c6cc76744-s2pww\" (UID: \"072395f0-b9b7-438a-83d6-9f92feb714ca\") " pod="openshift-console/console-7c6cc76744-s2pww" Oct 30 17:12:41.398066 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.398053 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/072395f0-b9b7-438a-83d6-9f92feb714ca-console-serving-cert\") pod \"console-7c6cc76744-s2pww\" (UID: \"072395f0-b9b7-438a-83d6-9f92feb714ca\") " pod="openshift-console/console-7c6cc76744-s2pww" Oct 30 17:12:41.418846 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.418817 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdc9z\" (UniqueName: \"kubernetes.io/projected/072395f0-b9b7-438a-83d6-9f92feb714ca-kube-api-access-bdc9z\") pod \"console-7c6cc76744-s2pww\" (UID: \"072395f0-b9b7-438a-83d6-9f92feb714ca\") " pod="openshift-console/console-7c6cc76744-s2pww" Oct 30 17:12:41.464539 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.464509 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6cc76744-s2pww" Oct 30 17:12:41.731231 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.731200 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-stqrx_2664a881-44d7-470a-93d0-df1b3eaca0bc/node-ca/0.log" Oct 30 17:12:41.771338 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.771312 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c6cc76744-s2pww"] Oct 30 17:12:41.774430 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:41.774392 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod072395f0_b9b7_438a_83d6_9f92feb714ca.slice/crio-182e060d885520c5fee2241143c8b2bc5993410d7e510d97c7796afb5792b394 WatchSource:0}: Error finding container 182e060d885520c5fee2241143c8b2bc5993410d7e510d97c7796afb5792b394: Status 404 returned error can't find the container with id 182e060d885520c5fee2241143c8b2bc5993410d7e510d97c7796afb5792b394 Oct 30 17:12:41.944319 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.944285 4965 patch_prober.go:29] interesting pod/image-registry-77579944f4-hctng container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 17:12:41.944485 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:41.944341 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-77579944f4-hctng" podUID="fd74ba94-86d0-4707-94e5-6822c4ab5751" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:12:42.131692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:42.131660 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-ddb54c4f7-54tw2_924ac066-ff98-4320-9eb2-268f0a39ca33/router/0.log" Oct 30 17:12:42.298895 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:42.298836 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6cc76744-s2pww" event={"ID":"072395f0-b9b7-438a-83d6-9f92feb714ca","Type":"ContainerStarted","Data":"182e060d885520c5fee2241143c8b2bc5993410d7e510d97c7796afb5792b394"} Oct 30 17:12:42.332172 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:42.332131 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rdjn5_89eff971-c01b-4b06-9c16-4332956558a3/serve-healthcheck-canary/0.log" Oct 30 17:12:42.532555 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:42.532485 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-m65sr_46515ca5-213a-4acc-8db2-680096edc147/migrator/0.log" Oct 30 17:12:42.731832 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:42.731796 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-s7vbz_0ea73d05-f4cd-45a4-b043-d0ec3391235e/migrator/0.log" Oct 30 17:12:42.932951 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:42.932919 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-jccch_27ac623c-2f57-44ec-a1b1-3ad87909452d/kube-storage-version-migrator-operator/0.log" Oct 30 17:12:45.092092 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:45.091683 4965 scope.go:122] "RemoveContainer" containerID="c654105b733943dee5c08c0eabc25079203957327c1481ce32a4177931ed4bea" Oct 30 17:12:45.317943 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:45.317901 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4w8fl" event={"ID":"16cd937e-4c6d-45d0-a9de-f77dc836d082","Type":"ContainerStarted","Data":"46039aef694f9a2e68ee8eeba42b39b1172611b28791965f1c9bb17a57d29a63"} Oct 30 17:12:45.319955 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:45.319920 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6cc76744-s2pww" event={"ID":"072395f0-b9b7-438a-83d6-9f92feb714ca","Type":"ContainerStarted","Data":"ebf4c3cd1e7a642850abebae1f3dbb9e63335e95212ac6a11151859b7f826407"} Oct 30 17:12:45.403830 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:45.403719 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7c6cc76744-s2pww" podStartSLOduration=1.328979516 podStartE2EDuration="4.403699283s" totalImagesPullingTime="3.074719767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:45.402921481 +0000 UTC m=+23.455021437" watchObservedRunningTime="2026-10-30 17:12:45.403699283 +0000 UTC m=+23.455799216" Oct 30 17:12:46.695032 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:46.695002 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:12:47.587032 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:47.587000 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jnfks" Oct 30 17:12:51.177101 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:51.177065 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-t2pqm" Oct 30 17:12:51.464995 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:51.464908 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c6cc76744-s2pww" Oct 30 17:12:51.464995 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:51.464979 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c6cc76744-s2pww" Oct 30 17:12:51.471819 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:51.471789 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c6cc76744-s2pww" Oct 30 17:12:51.944937 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:51.944902 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:12:52.357263 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:52.357229 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c6cc76744-s2pww" Oct 30 17:12:53.445969 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:53.445936 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c6cc76744-s2pww"] Oct 30 17:12:54.364507 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:54.364459 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-j6pg9" event={"ID":"99c91f4b-d893-45d0-82a0-abbb240a49e7","Type":"ContainerStarted","Data":"0c659461e2cfda3dde7c188f9cc4bd51b565e167c552f26860819877e909b99f"} Oct 30 17:12:54.364702 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:54.364530 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-j6pg9" Oct 30 17:12:54.378334 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:54.378305 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-j6pg9" Oct 30 17:12:54.451260 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:54.451195 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-j6pg9" podStartSLOduration=1.9512797339999999 podStartE2EDuration="18.451179272s" totalImagesPullingTime="16.499899538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:54.450088804 +0000 UTC m=+32.502188753" watchObservedRunningTime="2026-10-30 17:12:54.451179272 +0000 UTC m=+32.503279202" Oct 30 17:12:57.014883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.014834 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bh2zb"] Oct 30 17:12:57.049779 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.049737 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bh2zb" Oct 30 17:12:57.086423 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.086383 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 17:12:57.086584 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.086438 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 17:12:57.086584 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.086453 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 17:12:57.086584 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.086471 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-726zt\"" Oct 30 17:12:57.086584 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.086513 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 17:12:57.138727 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.138692 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/066821ed-2f29-44e7-924d-00ac9d0d6273-node-exporter-wtmp\") pod \"node-exporter-bh2zb\" (UID: \"066821ed-2f29-44e7-924d-00ac9d0d6273\") " pod="openshift-monitoring/node-exporter-bh2zb" Oct 30 17:12:57.138727 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.138731 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/066821ed-2f29-44e7-924d-00ac9d0d6273-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bh2zb\" (UID: \"066821ed-2f29-44e7-924d-00ac9d0d6273\") " pod="openshift-monitoring/node-exporter-bh2zb" Oct 30 17:12:57.138993 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.138973 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/066821ed-2f29-44e7-924d-00ac9d0d6273-node-exporter-accelerators-collector-config\") pod \"node-exporter-bh2zb\" (UID: \"066821ed-2f29-44e7-924d-00ac9d0d6273\") " pod="openshift-monitoring/node-exporter-bh2zb" Oct 30 17:12:57.139045 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.139018 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf22q\" (UniqueName: \"kubernetes.io/projected/066821ed-2f29-44e7-924d-00ac9d0d6273-kube-api-access-cf22q\") pod \"node-exporter-bh2zb\" (UID: \"066821ed-2f29-44e7-924d-00ac9d0d6273\") " pod="openshift-monitoring/node-exporter-bh2zb" Oct 30 17:12:57.139097 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.139063 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/066821ed-2f29-44e7-924d-00ac9d0d6273-node-exporter-textfile\") pod \"node-exporter-bh2zb\" (UID: \"066821ed-2f29-44e7-924d-00ac9d0d6273\") " pod="openshift-monitoring/node-exporter-bh2zb" Oct 30 17:12:57.139097 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.139085 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/066821ed-2f29-44e7-924d-00ac9d0d6273-node-exporter-tls\") pod \"node-exporter-bh2zb\" (UID: \"066821ed-2f29-44e7-924d-00ac9d0d6273\") " pod="openshift-monitoring/node-exporter-bh2zb" Oct 30 17:12:57.139195 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.139177 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/066821ed-2f29-44e7-924d-00ac9d0d6273-root\") pod \"node-exporter-bh2zb\" (UID: \"066821ed-2f29-44e7-924d-00ac9d0d6273\") " pod="openshift-monitoring/node-exporter-bh2zb" Oct 30 17:12:57.139254 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.139237 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/066821ed-2f29-44e7-924d-00ac9d0d6273-sys\") pod \"node-exporter-bh2zb\" (UID: \"066821ed-2f29-44e7-924d-00ac9d0d6273\") " pod="openshift-monitoring/node-exporter-bh2zb" Oct 30 17:12:57.139304 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.139274 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/066821ed-2f29-44e7-924d-00ac9d0d6273-metrics-client-ca\") pod \"node-exporter-bh2zb\" (UID: \"066821ed-2f29-44e7-924d-00ac9d0d6273\") " pod="openshift-monitoring/node-exporter-bh2zb" Oct 30 17:12:57.241157 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.240379 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/066821ed-2f29-44e7-924d-00ac9d0d6273-node-exporter-accelerators-collector-config\") pod \"node-exporter-bh2zb\" (UID: \"066821ed-2f29-44e7-924d-00ac9d0d6273\") " pod="openshift-monitoring/node-exporter-bh2zb" Oct 30 17:12:57.241157 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.240430 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf22q\" (UniqueName: \"kubernetes.io/projected/066821ed-2f29-44e7-924d-00ac9d0d6273-kube-api-access-cf22q\") pod \"node-exporter-bh2zb\" (UID: \"066821ed-2f29-44e7-924d-00ac9d0d6273\") " pod="openshift-monitoring/node-exporter-bh2zb" Oct 30 17:12:57.241157 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.240472 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/066821ed-2f29-44e7-924d-00ac9d0d6273-node-exporter-textfile\") pod \"node-exporter-bh2zb\" (UID: \"066821ed-2f29-44e7-924d-00ac9d0d6273\") " pod="openshift-monitoring/node-exporter-bh2zb" Oct 30 17:12:57.241157 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.240497 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/066821ed-2f29-44e7-924d-00ac9d0d6273-node-exporter-tls\") pod \"node-exporter-bh2zb\" (UID: \"066821ed-2f29-44e7-924d-00ac9d0d6273\") " pod="openshift-monitoring/node-exporter-bh2zb" Oct 30 17:12:57.241157 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.240532 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/066821ed-2f29-44e7-924d-00ac9d0d6273-root\") pod \"node-exporter-bh2zb\" (UID: \"066821ed-2f29-44e7-924d-00ac9d0d6273\") " pod="openshift-monitoring/node-exporter-bh2zb" Oct 30 17:12:57.241157 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.240560 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/066821ed-2f29-44e7-924d-00ac9d0d6273-sys\") pod \"node-exporter-bh2zb\" (UID: \"066821ed-2f29-44e7-924d-00ac9d0d6273\") " pod="openshift-monitoring/node-exporter-bh2zb" Oct 30 17:12:57.241157 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.240576 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/066821ed-2f29-44e7-924d-00ac9d0d6273-metrics-client-ca\") pod \"node-exporter-bh2zb\" (UID: \"066821ed-2f29-44e7-924d-00ac9d0d6273\") " pod="openshift-monitoring/node-exporter-bh2zb" Oct 30 17:12:57.241157 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.240613 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/066821ed-2f29-44e7-924d-00ac9d0d6273-node-exporter-wtmp\") pod \"node-exporter-bh2zb\" (UID: \"066821ed-2f29-44e7-924d-00ac9d0d6273\") " pod="openshift-monitoring/node-exporter-bh2zb" Oct 30 17:12:57.241157 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.240642 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/066821ed-2f29-44e7-924d-00ac9d0d6273-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bh2zb\" (UID: \"066821ed-2f29-44e7-924d-00ac9d0d6273\") " pod="openshift-monitoring/node-exporter-bh2zb" Oct 30 17:12:57.242906 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.242877 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/066821ed-2f29-44e7-924d-00ac9d0d6273-root\") pod \"node-exporter-bh2zb\" (UID: \"066821ed-2f29-44e7-924d-00ac9d0d6273\") " pod="openshift-monitoring/node-exporter-bh2zb" Oct 30 17:12:57.243009 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.242951 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/066821ed-2f29-44e7-924d-00ac9d0d6273-sys\") pod \"node-exporter-bh2zb\" (UID: \"066821ed-2f29-44e7-924d-00ac9d0d6273\") " pod="openshift-monitoring/node-exporter-bh2zb" Oct 30 17:12:57.243575 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.243544 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/066821ed-2f29-44e7-924d-00ac9d0d6273-metrics-client-ca\") pod \"node-exporter-bh2zb\" (UID: \"066821ed-2f29-44e7-924d-00ac9d0d6273\") " pod="openshift-monitoring/node-exporter-bh2zb" Oct 30 17:12:57.243980 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.243683 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/066821ed-2f29-44e7-924d-00ac9d0d6273-node-exporter-wtmp\") pod \"node-exporter-bh2zb\" (UID: \"066821ed-2f29-44e7-924d-00ac9d0d6273\") " pod="openshift-monitoring/node-exporter-bh2zb" Oct 30 17:12:57.243980 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.243696 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/066821ed-2f29-44e7-924d-00ac9d0d6273-node-exporter-accelerators-collector-config\") pod \"node-exporter-bh2zb\" (UID: \"066821ed-2f29-44e7-924d-00ac9d0d6273\") " pod="openshift-monitoring/node-exporter-bh2zb" Oct 30 17:12:57.243980 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.243935 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/066821ed-2f29-44e7-924d-00ac9d0d6273-node-exporter-textfile\") pod \"node-exporter-bh2zb\" (UID: \"066821ed-2f29-44e7-924d-00ac9d0d6273\") " pod="openshift-monitoring/node-exporter-bh2zb" Oct 30 17:12:57.245172 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.245126 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/066821ed-2f29-44e7-924d-00ac9d0d6273-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bh2zb\" (UID: \"066821ed-2f29-44e7-924d-00ac9d0d6273\") " pod="openshift-monitoring/node-exporter-bh2zb" Oct 30 17:12:57.247516 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.246818 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/066821ed-2f29-44e7-924d-00ac9d0d6273-node-exporter-tls\") pod \"node-exporter-bh2zb\" (UID: \"066821ed-2f29-44e7-924d-00ac9d0d6273\") " pod="openshift-monitoring/node-exporter-bh2zb" Oct 30 17:12:57.266608 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.266557 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf22q\" (UniqueName: \"kubernetes.io/projected/066821ed-2f29-44e7-924d-00ac9d0d6273-kube-api-access-cf22q\") pod \"node-exporter-bh2zb\" (UID: \"066821ed-2f29-44e7-924d-00ac9d0d6273\") " pod="openshift-monitoring/node-exporter-bh2zb" Oct 30 17:12:57.437847 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:57.437806 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bh2zb" Oct 30 17:12:57.482187 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:12:57.482146 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod066821ed_2f29_44e7_924d_00ac9d0d6273.slice/crio-0d2bebdc7d520abcdaa6f3b3cfc56483798af90644cfb02c8e171075007cc229 WatchSource:0}: Error finding container 0d2bebdc7d520abcdaa6f3b3cfc56483798af90644cfb02c8e171075007cc229: Status 404 returned error can't find the container with id 0d2bebdc7d520abcdaa6f3b3cfc56483798af90644cfb02c8e171075007cc229 Oct 30 17:12:58.391140 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:58.391096 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bh2zb" event={"ID":"066821ed-2f29-44e7-924d-00ac9d0d6273","Type":"ContainerStarted","Data":"0d2bebdc7d520abcdaa6f3b3cfc56483798af90644cfb02c8e171075007cc229"} Oct 30 17:12:59.290029 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:59.289441 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-65f579c9f9-fhj5r" Oct 30 17:12:59.389807 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:59.389691 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-77579944f4-hctng"] Oct 30 17:12:59.416047 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:12:59.410252 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bh2zb" event={"ID":"066821ed-2f29-44e7-924d-00ac9d0d6273","Type":"ContainerStarted","Data":"04a3eb1533d9af16cd2970147274735e0b11cf3126cba18d03ebb49718677ebe"} Oct 30 17:13:00.446481 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:00.446407 4965 generic.go:356] "Generic (PLEG): container finished" podID="066821ed-2f29-44e7-924d-00ac9d0d6273" containerID="04a3eb1533d9af16cd2970147274735e0b11cf3126cba18d03ebb49718677ebe" exitCode=0 Oct 30 17:13:00.446481 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:00.446462 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bh2zb" event={"ID":"066821ed-2f29-44e7-924d-00ac9d0d6273","Type":"ContainerDied","Data":"04a3eb1533d9af16cd2970147274735e0b11cf3126cba18d03ebb49718677ebe"} Oct 30 17:13:01.454196 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:01.454154 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bh2zb" event={"ID":"066821ed-2f29-44e7-924d-00ac9d0d6273","Type":"ContainerStarted","Data":"7527855732b2c1c8e11cf3edb955feab4178f682ab9eccf26494b1b502c0bb07"} Oct 30 17:13:01.454687 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:01.454226 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bh2zb" event={"ID":"066821ed-2f29-44e7-924d-00ac9d0d6273","Type":"ContainerStarted","Data":"619d20235c0586c744b201f5d6512142ae10cac45f66f4d4de3777c3539902f6"} Oct 30 17:13:01.525472 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:01.525415 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bh2zb" podStartSLOduration=3.730836677 podStartE2EDuration="5.525399291s" totalImagesPullingTime="1.655648223s" totalInitContainerRuntime="138.914391ms" isStatefulPod=true podCreationTimestamp="2026-10-30 17:12:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:01.512855673 +0000 UTC m=+39.564955605" watchObservedRunningTime="2026-10-30 17:13:01.525399291 +0000 UTC m=+39.577499245" Oct 30 17:13:01.711530 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:01.711209 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" podUID="b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce" containerName="registry" containerID="cri-o://d2d17cf7af9a1aabade19d16f36f5bf7411cbbaceadf0763c677de2b9e050ce5" gracePeriod=30 Oct 30 17:13:01.995633 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:01.995598 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:13:02.104685 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.104652 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-ca-trust-extracted\") pod \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " Oct 30 17:13:02.104849 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.104766 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-trusted-ca\") pod \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " Oct 30 17:13:02.104849 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.104795 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-kube-api-access-brgv8\" (UniqueName: \"kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-kube-api-access-brgv8\") pod \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " Oct 30 17:13:02.104849 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.104843 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-image-registry-private-configuration\") pod \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " Oct 30 17:13:02.105048 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.104906 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-installation-pull-secrets\") pod \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " Oct 30 17:13:02.105048 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.104939 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-certificates\") pod \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " Oct 30 17:13:02.105048 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.104970 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-tls\" (UniqueName: \"kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-tls\") pod \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " Oct 30 17:13:02.105048 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.105014 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-bound-sa-token\") pod \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\" (UID: \"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce\") " Oct 30 17:13:02.105278 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.105249 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-trusted-ca" pod "b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce" (UID: "b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:02.106274 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.106199 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-certificates" pod "b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce" (UID: "b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:02.109551 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.109514 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-image-registry-private-configuration" pod "b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce" (UID: "b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:02.110253 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.110216 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-installation-pull-secrets" pod "b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce" (UID: "b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:02.110376 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.110338 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-bound-sa-token" pod "b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce" (UID: "b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:02.110502 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.110473 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-kube-api-access-brgv8" pod "b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce" (UID: "b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce"). InnerVolumeSpecName "kube-api-access-brgv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:02.111917 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.111854 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-tls" pod "b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce" (UID: "b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:02.113173 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.113146 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-ca-trust-extracted" pod "b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce" (UID: "b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:02.205783 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.205743 4965 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-ca-trust-extracted\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:13:02.205783 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.205781 4965 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-trusted-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:13:02.206050 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.205797 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brgv8\" (UniqueName: \"kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-kube-api-access-brgv8\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:13:02.206050 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.205815 4965 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-image-registry-private-configuration\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:13:02.206050 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.205830 4965 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-installation-pull-secrets\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:13:02.206050 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.205841 4965 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-certificates\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:13:02.206050 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.205849 4965 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-registry-tls\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:13:02.206050 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.205887 4965 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce-bound-sa-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:13:02.459850 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.459810 4965 generic.go:356] "Generic (PLEG): container finished" podID="b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce" containerID="d2d17cf7af9a1aabade19d16f36f5bf7411cbbaceadf0763c677de2b9e050ce5" exitCode=0 Oct 30 17:13:02.460274 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.459897 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" event={"ID":"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce","Type":"ContainerDied","Data":"d2d17cf7af9a1aabade19d16f36f5bf7411cbbaceadf0763c677de2b9e050ce5"} Oct 30 17:13:02.460274 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.459922 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" Oct 30 17:13:02.460274 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.459951 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-585b9dc6c5-dblkr" event={"ID":"b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce","Type":"ContainerDied","Data":"1db9d57d93616d600cb967e4161c6118f8991004bdfda9c4b38a8f706776ba0d"} Oct 30 17:13:02.460274 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.459971 4965 scope.go:122] "RemoveContainer" containerID="d2d17cf7af9a1aabade19d16f36f5bf7411cbbaceadf0763c677de2b9e050ce5" Oct 30 17:13:02.468892 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.468844 4965 scope.go:122] "RemoveContainer" containerID="d2d17cf7af9a1aabade19d16f36f5bf7411cbbaceadf0763c677de2b9e050ce5" Oct 30 17:13:02.469163 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:13:02.469140 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2d17cf7af9a1aabade19d16f36f5bf7411cbbaceadf0763c677de2b9e050ce5\": container with ID starting with d2d17cf7af9a1aabade19d16f36f5bf7411cbbaceadf0763c677de2b9e050ce5 not found: ID does not exist" containerID="d2d17cf7af9a1aabade19d16f36f5bf7411cbbaceadf0763c677de2b9e050ce5" Oct 30 17:13:02.469226 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.469173 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2d17cf7af9a1aabade19d16f36f5bf7411cbbaceadf0763c677de2b9e050ce5"} err="failed to get container status \"d2d17cf7af9a1aabade19d16f36f5bf7411cbbaceadf0763c677de2b9e050ce5\": rpc error: code = NotFound desc = could not find container \"d2d17cf7af9a1aabade19d16f36f5bf7411cbbaceadf0763c677de2b9e050ce5\": container with ID starting with d2d17cf7af9a1aabade19d16f36f5bf7411cbbaceadf0763c677de2b9e050ce5 not found: ID does not exist" Oct 30 17:13:02.511442 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.511408 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-585b9dc6c5-dblkr"] Oct 30 17:13:02.523663 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:02.523633 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-585b9dc6c5-dblkr"] Oct 30 17:13:04.098886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:04.098798 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce" path="/var/lib/kubelet/pods/b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce/volumes" Oct 30 17:13:05.995278 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:13:05.995242 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:13:08.975968 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:08.975931 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-fngsc-t5d6p"] Oct 30 17:13:08.976377 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:08.976143 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-fngsc-t5d6p" podUID="65986964-0b39-48e4-9163-705ed74afd18" containerName="insights-gathering" containerID="cri-o://d181ee8d25585aa0eea312407fa89fcb3f8df5c90f00363a3c3715b499c62817" gracePeriod=30 Oct 30 17:13:08.987007 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:08.986981 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-2ftd7-xc6xx"] Oct 30 17:13:08.987188 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:08.987169 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-2ftd7-xc6xx" podUID="13444fae-8e71-41ee-8489-55b8606492fb" containerName="insights-gathering" containerID="cri-o://7b0c8c13a33067c6eb1c2d86c8e6da418f167cf984bd042570c78809302d4d34" gracePeriod=30 Oct 30 17:13:09.296375 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.296342 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-2ftd7-xc6xx" Oct 30 17:13:09.312586 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.312557 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fngsc-t5d6p" Oct 30 17:13:09.375486 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.375446 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13444fae-8e71-41ee-8489-55b8606492fb-kube-api-access-drkzm\" (UniqueName: \"kubernetes.io/projected/13444fae-8e71-41ee-8489-55b8606492fb-kube-api-access-drkzm\") pod \"13444fae-8e71-41ee-8489-55b8606492fb\" (UID: \"13444fae-8e71-41ee-8489-55b8606492fb\") " Oct 30 17:13:09.375661 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.375546 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13444fae-8e71-41ee-8489-55b8606492fb-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/13444fae-8e71-41ee-8489-55b8606492fb-archives-path\") pod \"13444fae-8e71-41ee-8489-55b8606492fb\" (UID: \"13444fae-8e71-41ee-8489-55b8606492fb\") " Oct 30 17:13:09.375661 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.375586 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13444fae-8e71-41ee-8489-55b8606492fb-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13444fae-8e71-41ee-8489-55b8606492fb-trusted-ca-bundle\") pod \"13444fae-8e71-41ee-8489-55b8606492fb\" (UID: \"13444fae-8e71-41ee-8489-55b8606492fb\") " Oct 30 17:13:09.375661 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.375647 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13444fae-8e71-41ee-8489-55b8606492fb-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13444fae-8e71-41ee-8489-55b8606492fb-service-ca-bundle\") pod \"13444fae-8e71-41ee-8489-55b8606492fb\" (UID: \"13444fae-8e71-41ee-8489-55b8606492fb\") " Oct 30 17:13:09.375977 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.375947 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13444fae-8e71-41ee-8489-55b8606492fb-archives-path" pod "13444fae-8e71-41ee-8489-55b8606492fb" (UID: "13444fae-8e71-41ee-8489-55b8606492fb"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:09.376075 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.376028 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13444fae-8e71-41ee-8489-55b8606492fb-trusted-ca-bundle" pod "13444fae-8e71-41ee-8489-55b8606492fb" (UID: "13444fae-8e71-41ee-8489-55b8606492fb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:09.376075 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.376026 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13444fae-8e71-41ee-8489-55b8606492fb-service-ca-bundle" pod "13444fae-8e71-41ee-8489-55b8606492fb" (UID: "13444fae-8e71-41ee-8489-55b8606492fb"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:09.377841 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.377816 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13444fae-8e71-41ee-8489-55b8606492fb-kube-api-access-drkzm" pod "13444fae-8e71-41ee-8489-55b8606492fb" (UID: "13444fae-8e71-41ee-8489-55b8606492fb"). InnerVolumeSpecName "kube-api-access-drkzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:09.477096 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.477055 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65986964-0b39-48e4-9163-705ed74afd18-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65986964-0b39-48e4-9163-705ed74afd18-service-ca-bundle\") pod \"65986964-0b39-48e4-9163-705ed74afd18\" (UID: \"65986964-0b39-48e4-9163-705ed74afd18\") " Oct 30 17:13:09.477096 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.477100 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65986964-0b39-48e4-9163-705ed74afd18-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65986964-0b39-48e4-9163-705ed74afd18-trusted-ca-bundle\") pod \"65986964-0b39-48e4-9163-705ed74afd18\" (UID: \"65986964-0b39-48e4-9163-705ed74afd18\") " Oct 30 17:13:09.477330 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.477133 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65986964-0b39-48e4-9163-705ed74afd18-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/65986964-0b39-48e4-9163-705ed74afd18-archives-path\") pod \"65986964-0b39-48e4-9163-705ed74afd18\" (UID: \"65986964-0b39-48e4-9163-705ed74afd18\") " Oct 30 17:13:09.477330 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.477182 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65986964-0b39-48e4-9163-705ed74afd18-kube-api-access-tfznx\" (UniqueName: \"kubernetes.io/projected/65986964-0b39-48e4-9163-705ed74afd18-kube-api-access-tfznx\") pod \"65986964-0b39-48e4-9163-705ed74afd18\" (UID: \"65986964-0b39-48e4-9163-705ed74afd18\") " Oct 30 17:13:09.477440 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.477351 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drkzm\" (UniqueName: \"kubernetes.io/projected/13444fae-8e71-41ee-8489-55b8606492fb-kube-api-access-drkzm\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:13:09.477440 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.477363 4965 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/13444fae-8e71-41ee-8489-55b8606492fb-archives-path\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:13:09.477440 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.477373 4965 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13444fae-8e71-41ee-8489-55b8606492fb-trusted-ca-bundle\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:13:09.477440 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.477386 4965 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13444fae-8e71-41ee-8489-55b8606492fb-service-ca-bundle\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:13:09.477646 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.477473 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65986964-0b39-48e4-9163-705ed74afd18-service-ca-bundle" pod "65986964-0b39-48e4-9163-705ed74afd18" (UID: "65986964-0b39-48e4-9163-705ed74afd18"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:09.477646 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.477540 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65986964-0b39-48e4-9163-705ed74afd18-archives-path" pod "65986964-0b39-48e4-9163-705ed74afd18" (UID: "65986964-0b39-48e4-9163-705ed74afd18"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:09.477646 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.477550 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65986964-0b39-48e4-9163-705ed74afd18-trusted-ca-bundle" pod "65986964-0b39-48e4-9163-705ed74afd18" (UID: "65986964-0b39-48e4-9163-705ed74afd18"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:09.479334 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.479313 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65986964-0b39-48e4-9163-705ed74afd18-kube-api-access-tfznx" pod "65986964-0b39-48e4-9163-705ed74afd18" (UID: "65986964-0b39-48e4-9163-705ed74afd18"). InnerVolumeSpecName "kube-api-access-tfznx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:09.490587 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.490551 4965 generic.go:356] "Generic (PLEG): container finished" podID="65986964-0b39-48e4-9163-705ed74afd18" containerID="d181ee8d25585aa0eea312407fa89fcb3f8df5c90f00363a3c3715b499c62817" exitCode=2 Oct 30 17:13:09.490737 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.490602 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fngsc-t5d6p" Oct 30 17:13:09.490737 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.490599 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fngsc-t5d6p" event={"ID":"65986964-0b39-48e4-9163-705ed74afd18","Type":"ContainerDied","Data":"d181ee8d25585aa0eea312407fa89fcb3f8df5c90f00363a3c3715b499c62817"} Oct 30 17:13:09.490737 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.490664 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fngsc-t5d6p" event={"ID":"65986964-0b39-48e4-9163-705ed74afd18","Type":"ContainerDied","Data":"4ff35bbc71b9691038fdb167644c1d6d8496215f3c709f8dd8c7267de461984f"} Oct 30 17:13:09.490737 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.490686 4965 scope.go:122] "RemoveContainer" containerID="d181ee8d25585aa0eea312407fa89fcb3f8df5c90f00363a3c3715b499c62817" Oct 30 17:13:09.492082 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.492058 4965 generic.go:356] "Generic (PLEG): container finished" podID="13444fae-8e71-41ee-8489-55b8606492fb" containerID="7b0c8c13a33067c6eb1c2d86c8e6da418f167cf984bd042570c78809302d4d34" exitCode=2 Oct 30 17:13:09.492193 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.492111 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-2ftd7-xc6xx" Oct 30 17:13:09.492193 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.492136 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-2ftd7-xc6xx" event={"ID":"13444fae-8e71-41ee-8489-55b8606492fb","Type":"ContainerDied","Data":"7b0c8c13a33067c6eb1c2d86c8e6da418f167cf984bd042570c78809302d4d34"} Oct 30 17:13:09.492193 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.492186 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-2ftd7-xc6xx" event={"ID":"13444fae-8e71-41ee-8489-55b8606492fb","Type":"ContainerDied","Data":"837040638b1b42bf20445cb551dc04c6277f9fbef3a55987538fc7f0b9bf8c15"} Oct 30 17:13:09.499763 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.499744 4965 scope.go:122] "RemoveContainer" containerID="d181ee8d25585aa0eea312407fa89fcb3f8df5c90f00363a3c3715b499c62817" Oct 30 17:13:09.500103 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:13:09.500083 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d181ee8d25585aa0eea312407fa89fcb3f8df5c90f00363a3c3715b499c62817\": container with ID starting with d181ee8d25585aa0eea312407fa89fcb3f8df5c90f00363a3c3715b499c62817 not found: ID does not exist" containerID="d181ee8d25585aa0eea312407fa89fcb3f8df5c90f00363a3c3715b499c62817" Oct 30 17:13:09.500179 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.500116 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d181ee8d25585aa0eea312407fa89fcb3f8df5c90f00363a3c3715b499c62817"} err="failed to get container status \"d181ee8d25585aa0eea312407fa89fcb3f8df5c90f00363a3c3715b499c62817\": rpc error: code = NotFound desc = could not find container \"d181ee8d25585aa0eea312407fa89fcb3f8df5c90f00363a3c3715b499c62817\": container with ID starting with d181ee8d25585aa0eea312407fa89fcb3f8df5c90f00363a3c3715b499c62817 not found: ID does not exist" Oct 30 17:13:09.500179 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.500136 4965 scope.go:122] "RemoveContainer" containerID="7b0c8c13a33067c6eb1c2d86c8e6da418f167cf984bd042570c78809302d4d34" Oct 30 17:13:09.508703 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.508684 4965 scope.go:122] "RemoveContainer" containerID="7b0c8c13a33067c6eb1c2d86c8e6da418f167cf984bd042570c78809302d4d34" Oct 30 17:13:09.509018 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:13:09.508998 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b0c8c13a33067c6eb1c2d86c8e6da418f167cf984bd042570c78809302d4d34\": container with ID starting with 7b0c8c13a33067c6eb1c2d86c8e6da418f167cf984bd042570c78809302d4d34 not found: ID does not exist" containerID="7b0c8c13a33067c6eb1c2d86c8e6da418f167cf984bd042570c78809302d4d34" Oct 30 17:13:09.509071 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.509029 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b0c8c13a33067c6eb1c2d86c8e6da418f167cf984bd042570c78809302d4d34"} err="failed to get container status \"7b0c8c13a33067c6eb1c2d86c8e6da418f167cf984bd042570c78809302d4d34\": rpc error: code = NotFound desc = could not find container \"7b0c8c13a33067c6eb1c2d86c8e6da418f167cf984bd042570c78809302d4d34\": container with ID starting with 7b0c8c13a33067c6eb1c2d86c8e6da418f167cf984bd042570c78809302d4d34 not found: ID does not exist" Oct 30 17:13:09.542325 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.542247 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-fngsc-t5d6p"] Oct 30 17:13:09.546539 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.546513 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-fngsc-t5d6p"] Oct 30 17:13:09.578247 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.578211 4965 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65986964-0b39-48e4-9163-705ed74afd18-service-ca-bundle\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:13:09.578247 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.578239 4965 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65986964-0b39-48e4-9163-705ed74afd18-trusted-ca-bundle\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:13:09.578488 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.578264 4965 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/65986964-0b39-48e4-9163-705ed74afd18-archives-path\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:13:09.578488 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.578277 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfznx\" (UniqueName: \"kubernetes.io/projected/65986964-0b39-48e4-9163-705ed74afd18-kube-api-access-tfznx\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:13:09.584668 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.584638 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-2ftd7-xc6xx"] Oct 30 17:13:09.586334 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:09.586310 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-2ftd7-xc6xx"] Oct 30 17:13:10.097184 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:10.097155 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13444fae-8e71-41ee-8489-55b8606492fb" path="/var/lib/kubelet/pods/13444fae-8e71-41ee-8489-55b8606492fb/volumes" Oct 30 17:13:10.097602 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:10.097527 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65986964-0b39-48e4-9163-705ed74afd18" path="/var/lib/kubelet/pods/65986964-0b39-48e4-9163-705ed74afd18/volumes" Oct 30 17:13:19.387329 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:19.387271 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7c6cc76744-s2pww" podUID="072395f0-b9b7-438a-83d6-9f92feb714ca" containerName="console" containerID="cri-o://ebf4c3cd1e7a642850abebae1f3dbb9e63335e95212ac6a11151859b7f826407" gracePeriod=15 Oct 30 17:13:19.538060 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:19.538026 4965 generic.go:356] "Generic (PLEG): container finished" podID="072395f0-b9b7-438a-83d6-9f92feb714ca" containerID="ebf4c3cd1e7a642850abebae1f3dbb9e63335e95212ac6a11151859b7f826407" exitCode=0 Oct 30 17:13:19.538223 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:19.538111 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6cc76744-s2pww" event={"ID":"072395f0-b9b7-438a-83d6-9f92feb714ca","Type":"ContainerDied","Data":"ebf4c3cd1e7a642850abebae1f3dbb9e63335e95212ac6a11151859b7f826407"} Oct 30 17:13:19.632046 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:19.632023 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6cc76744-s2pww" Oct 30 17:13:19.666082 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:19.666020 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/072395f0-b9b7-438a-83d6-9f92feb714ca-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/072395f0-b9b7-438a-83d6-9f92feb714ca-oauth-serving-cert\") pod \"072395f0-b9b7-438a-83d6-9f92feb714ca\" (UID: \"072395f0-b9b7-438a-83d6-9f92feb714ca\") " Oct 30 17:13:19.666082 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:19.666067 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/072395f0-b9b7-438a-83d6-9f92feb714ca-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/072395f0-b9b7-438a-83d6-9f92feb714ca-console-oauth-config\") pod \"072395f0-b9b7-438a-83d6-9f92feb714ca\" (UID: \"072395f0-b9b7-438a-83d6-9f92feb714ca\") " Oct 30 17:13:19.666265 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:19.666097 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/072395f0-b9b7-438a-83d6-9f92feb714ca-kube-api-access-bdc9z\" (UniqueName: \"kubernetes.io/projected/072395f0-b9b7-438a-83d6-9f92feb714ca-kube-api-access-bdc9z\") pod \"072395f0-b9b7-438a-83d6-9f92feb714ca\" (UID: \"072395f0-b9b7-438a-83d6-9f92feb714ca\") " Oct 30 17:13:19.666265 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:19.666119 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/072395f0-b9b7-438a-83d6-9f92feb714ca-service-ca\" (UniqueName: \"kubernetes.io/configmap/072395f0-b9b7-438a-83d6-9f92feb714ca-service-ca\") pod \"072395f0-b9b7-438a-83d6-9f92feb714ca\" (UID: \"072395f0-b9b7-438a-83d6-9f92feb714ca\") " Oct 30 17:13:19.666265 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:19.666152 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/072395f0-b9b7-438a-83d6-9f92feb714ca-console-config\" (UniqueName: \"kubernetes.io/configmap/072395f0-b9b7-438a-83d6-9f92feb714ca-console-config\") pod \"072395f0-b9b7-438a-83d6-9f92feb714ca\" (UID: \"072395f0-b9b7-438a-83d6-9f92feb714ca\") " Oct 30 17:13:19.666265 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:19.666211 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/072395f0-b9b7-438a-83d6-9f92feb714ca-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/072395f0-b9b7-438a-83d6-9f92feb714ca-console-serving-cert\") pod \"072395f0-b9b7-438a-83d6-9f92feb714ca\" (UID: \"072395f0-b9b7-438a-83d6-9f92feb714ca\") " Oct 30 17:13:19.666536 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:19.666510 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/072395f0-b9b7-438a-83d6-9f92feb714ca-oauth-serving-cert" pod "072395f0-b9b7-438a-83d6-9f92feb714ca" (UID: "072395f0-b9b7-438a-83d6-9f92feb714ca"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:19.666608 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:19.666548 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/072395f0-b9b7-438a-83d6-9f92feb714ca-service-ca" pod "072395f0-b9b7-438a-83d6-9f92feb714ca" (UID: "072395f0-b9b7-438a-83d6-9f92feb714ca"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:19.666668 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:19.666620 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/072395f0-b9b7-438a-83d6-9f92feb714ca-console-config" pod "072395f0-b9b7-438a-83d6-9f92feb714ca" (UID: "072395f0-b9b7-438a-83d6-9f92feb714ca"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:19.668397 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:19.668372 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/072395f0-b9b7-438a-83d6-9f92feb714ca-console-serving-cert" pod "072395f0-b9b7-438a-83d6-9f92feb714ca" (UID: "072395f0-b9b7-438a-83d6-9f92feb714ca"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:19.668557 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:19.668418 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/072395f0-b9b7-438a-83d6-9f92feb714ca-kube-api-access-bdc9z" pod "072395f0-b9b7-438a-83d6-9f92feb714ca" (UID: "072395f0-b9b7-438a-83d6-9f92feb714ca"). InnerVolumeSpecName "kube-api-access-bdc9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:19.668557 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:19.668423 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/072395f0-b9b7-438a-83d6-9f92feb714ca-console-oauth-config" pod "072395f0-b9b7-438a-83d6-9f92feb714ca" (UID: "072395f0-b9b7-438a-83d6-9f92feb714ca"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:19.766688 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:19.766649 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/072395f0-b9b7-438a-83d6-9f92feb714ca-tmp\" (UniqueName: \"kubernetes.io/empty-dir/072395f0-b9b7-438a-83d6-9f92feb714ca-tmp\") pod \"072395f0-b9b7-438a-83d6-9f92feb714ca\" (UID: \"072395f0-b9b7-438a-83d6-9f92feb714ca\") " Oct 30 17:13:19.766851 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:19.766823 4965 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/072395f0-b9b7-438a-83d6-9f92feb714ca-console-config\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:13:19.766851 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:19.766837 4965 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/072395f0-b9b7-438a-83d6-9f92feb714ca-console-serving-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:13:19.766851 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:19.766845 4965 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/072395f0-b9b7-438a-83d6-9f92feb714ca-oauth-serving-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:13:19.766983 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:19.766854 4965 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/072395f0-b9b7-438a-83d6-9f92feb714ca-console-oauth-config\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:13:19.766983 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:19.766886 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdc9z\" (UniqueName: \"kubernetes.io/projected/072395f0-b9b7-438a-83d6-9f92feb714ca-kube-api-access-bdc9z\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:13:19.766983 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:19.766900 4965 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/072395f0-b9b7-438a-83d6-9f92feb714ca-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:13:19.766983 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:19.766934 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/072395f0-b9b7-438a-83d6-9f92feb714ca-tmp" pod "072395f0-b9b7-438a-83d6-9f92feb714ca" (UID: "072395f0-b9b7-438a-83d6-9f92feb714ca"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:19.868137 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:19.868103 4965 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/072395f0-b9b7-438a-83d6-9f92feb714ca-tmp\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:13:20.543770 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:20.543688 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6cc76744-s2pww" event={"ID":"072395f0-b9b7-438a-83d6-9f92feb714ca","Type":"ContainerDied","Data":"182e060d885520c5fee2241143c8b2bc5993410d7e510d97c7796afb5792b394"} Oct 30 17:13:20.543770 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:20.543715 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6cc76744-s2pww" Oct 30 17:13:20.543770 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:20.543743 4965 scope.go:122] "RemoveContainer" containerID="ebf4c3cd1e7a642850abebae1f3dbb9e63335e95212ac6a11151859b7f826407" Oct 30 17:13:20.597650 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:20.597620 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c6cc76744-s2pww"] Oct 30 17:13:20.609899 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:20.609853 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c6cc76744-s2pww"] Oct 30 17:13:22.097114 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:22.097087 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="072395f0-b9b7-438a-83d6-9f92feb714ca" path="/var/lib/kubelet/pods/072395f0-b9b7-438a-83d6-9f92feb714ca/volumes" Oct 30 17:13:24.434423 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:24.434381 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-77579944f4-hctng" podUID="fd74ba94-86d0-4707-94e5-6822c4ab5751" containerName="registry" containerID="cri-o://2241b433a010f7fa0c5a58ba334083de72fe0db0ac3af0e4a65b6aaec9b26672" gracePeriod=30 Oct 30 17:13:24.562426 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:24.562395 4965 generic.go:356] "Generic (PLEG): container finished" podID="fd74ba94-86d0-4707-94e5-6822c4ab5751" containerID="2241b433a010f7fa0c5a58ba334083de72fe0db0ac3af0e4a65b6aaec9b26672" exitCode=0 Oct 30 17:13:24.562571 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:24.562463 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77579944f4-hctng" event={"ID":"fd74ba94-86d0-4707-94e5-6822c4ab5751","Type":"ContainerDied","Data":"2241b433a010f7fa0c5a58ba334083de72fe0db0ac3af0e4a65b6aaec9b26672"} Oct 30 17:13:24.710955 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:24.710891 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:13:24.808596 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:24.808569 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd74ba94-86d0-4707-94e5-6822c4ab5751-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd74ba94-86d0-4707-94e5-6822c4ab5751-installation-pull-secrets\") pod \"fd74ba94-86d0-4707-94e5-6822c4ab5751\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " Oct 30 17:13:24.808741 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:24.808604 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd74ba94-86d0-4707-94e5-6822c4ab5751-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd74ba94-86d0-4707-94e5-6822c4ab5751-trusted-ca\") pod \"fd74ba94-86d0-4707-94e5-6822c4ab5751\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " Oct 30 17:13:24.808741 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:24.808645 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd74ba94-86d0-4707-94e5-6822c4ab5751-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd74ba94-86d0-4707-94e5-6822c4ab5751-image-registry-private-configuration\") pod \"fd74ba94-86d0-4707-94e5-6822c4ab5751\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " Oct 30 17:13:24.808741 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:24.808668 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-kube-api-access-qx4rg\" (UniqueName: \"kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-kube-api-access-qx4rg\") pod \"fd74ba94-86d0-4707-94e5-6822c4ab5751\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " Oct 30 17:13:24.808741 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:24.808698 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-bound-sa-token\") pod \"fd74ba94-86d0-4707-94e5-6822c4ab5751\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " Oct 30 17:13:24.808741 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:24.808719 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-certificates\") pod \"fd74ba94-86d0-4707-94e5-6822c4ab5751\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " Oct 30 17:13:24.809013 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:24.808751 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd74ba94-86d0-4707-94e5-6822c4ab5751-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd74ba94-86d0-4707-94e5-6822c4ab5751-ca-trust-extracted\") pod \"fd74ba94-86d0-4707-94e5-6822c4ab5751\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " Oct 30 17:13:24.809013 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:24.808789 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-tls\" (UniqueName: \"kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-tls\") pod \"fd74ba94-86d0-4707-94e5-6822c4ab5751\" (UID: \"fd74ba94-86d0-4707-94e5-6822c4ab5751\") " Oct 30 17:13:24.809170 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:24.809144 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd74ba94-86d0-4707-94e5-6822c4ab5751-trusted-ca" pod "fd74ba94-86d0-4707-94e5-6822c4ab5751" (UID: "fd74ba94-86d0-4707-94e5-6822c4ab5751"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:24.809308 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:24.809267 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-certificates" pod "fd74ba94-86d0-4707-94e5-6822c4ab5751" (UID: "fd74ba94-86d0-4707-94e5-6822c4ab5751"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:24.811262 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:24.811230 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-tls" pod "fd74ba94-86d0-4707-94e5-6822c4ab5751" (UID: "fd74ba94-86d0-4707-94e5-6822c4ab5751"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:24.811262 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:24.811258 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-kube-api-access-qx4rg" pod "fd74ba94-86d0-4707-94e5-6822c4ab5751" (UID: "fd74ba94-86d0-4707-94e5-6822c4ab5751"). InnerVolumeSpecName "kube-api-access-qx4rg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:24.811478 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:24.811450 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-bound-sa-token" pod "fd74ba94-86d0-4707-94e5-6822c4ab5751" (UID: "fd74ba94-86d0-4707-94e5-6822c4ab5751"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:24.811642 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:24.811625 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd74ba94-86d0-4707-94e5-6822c4ab5751-installation-pull-secrets" pod "fd74ba94-86d0-4707-94e5-6822c4ab5751" (UID: "fd74ba94-86d0-4707-94e5-6822c4ab5751"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:24.811788 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:24.811772 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd74ba94-86d0-4707-94e5-6822c4ab5751-image-registry-private-configuration" pod "fd74ba94-86d0-4707-94e5-6822c4ab5751" (UID: "fd74ba94-86d0-4707-94e5-6822c4ab5751"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:24.816161 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:24.816112 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd74ba94-86d0-4707-94e5-6822c4ab5751-ca-trust-extracted" pod "fd74ba94-86d0-4707-94e5-6822c4ab5751" (UID: "fd74ba94-86d0-4707-94e5-6822c4ab5751"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:24.909677 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:24.909642 4965 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd74ba94-86d0-4707-94e5-6822c4ab5751-image-registry-private-configuration\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:13:24.909677 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:24.909673 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx4rg\" (UniqueName: \"kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-kube-api-access-qx4rg\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:13:24.909677 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:24.909684 4965 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-bound-sa-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:13:24.909962 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:24.909694 4965 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-certificates\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:13:24.909962 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:24.909703 4965 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd74ba94-86d0-4707-94e5-6822c4ab5751-ca-trust-extracted\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:13:24.909962 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:24.909712 4965 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd74ba94-86d0-4707-94e5-6822c4ab5751-registry-tls\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:13:24.909962 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:24.909721 4965 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd74ba94-86d0-4707-94e5-6822c4ab5751-installation-pull-secrets\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:13:24.909962 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:24.909729 4965 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd74ba94-86d0-4707-94e5-6822c4ab5751-trusted-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:13:25.567469 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:25.567437 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77579944f4-hctng" event={"ID":"fd74ba94-86d0-4707-94e5-6822c4ab5751","Type":"ContainerDied","Data":"160da2f368e4265484e0e52507dfbffef970feac2caf859ba88689d6dd7bbf2e"} Oct 30 17:13:25.567469 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:25.567461 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77579944f4-hctng" Oct 30 17:13:25.568040 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:25.567499 4965 scope.go:122] "RemoveContainer" containerID="2241b433a010f7fa0c5a58ba334083de72fe0db0ac3af0e4a65b6aaec9b26672" Oct 30 17:13:25.630909 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:25.630882 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-77579944f4-hctng"] Oct 30 17:13:25.643085 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:25.643062 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-77579944f4-hctng"] Oct 30 17:13:26.096569 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:26.096533 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd74ba94-86d0-4707-94e5-6822c4ab5751" path="/var/lib/kubelet/pods/fd74ba94-86d0-4707-94e5-6822c4ab5751/volumes" Oct 30 17:13:35.996335 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:13:35.996293 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:13:52.673793 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:52.673756 4965 generic.go:356] "Generic (PLEG): container finished" podID="27ac623c-2f57-44ec-a1b1-3ad87909452d" containerID="c59e356e22f1cf78c847667f8dba99875135754baf2de47ada4ba1413447079d" exitCode=0 Oct 30 17:13:52.674262 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:52.673835 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jccch" event={"ID":"27ac623c-2f57-44ec-a1b1-3ad87909452d","Type":"ContainerDied","Data":"c59e356e22f1cf78c847667f8dba99875135754baf2de47ada4ba1413447079d"} Oct 30 17:13:52.674262 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:52.674234 4965 scope.go:122] "RemoveContainer" containerID="c59e356e22f1cf78c847667f8dba99875135754baf2de47ada4ba1413447079d" Oct 30 17:13:53.682348 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:53.682293 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jccch" event={"ID":"27ac623c-2f57-44ec-a1b1-3ad87909452d","Type":"ContainerStarted","Data":"9b06f7c7a748407c6ad1921390be0615eeb695683146abe1dddc205ed7d66351"} Oct 30 17:13:57.699584 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:57.699547 4965 generic.go:356] "Generic (PLEG): container finished" podID="195f952d-ee96-485d-99aa-8b47e6633c97" containerID="56d1a55ac073af357f593423a43c0b46cc7800e73d98aab7166b82638dc2d4cd" exitCode=0 Oct 30 17:13:57.700014 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:57.699623 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2tk8w" event={"ID":"195f952d-ee96-485d-99aa-8b47e6633c97","Type":"ContainerDied","Data":"56d1a55ac073af357f593423a43c0b46cc7800e73d98aab7166b82638dc2d4cd"} Oct 30 17:13:57.700057 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:57.700014 4965 scope.go:122] "RemoveContainer" containerID="56d1a55ac073af357f593423a43c0b46cc7800e73d98aab7166b82638dc2d4cd" Oct 30 17:13:58.705031 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:13:58.704991 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2tk8w" event={"ID":"195f952d-ee96-485d-99aa-8b47e6633c97","Type":"ContainerStarted","Data":"25d675955e8b9ab89910c6995d3a161ac9ed08517edd8577ac6e6ab75cb168e5"} Oct 30 17:14:05.995485 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:14:05.995448 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:14:35.995568 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:14:35.995486 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:15:05.994988 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:15:05.994946 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:15:35.994252 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:15:35.994217 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:16:05.995704 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:16:05.995620 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:16:35.994845 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:16:35.994808 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:17:05.994884 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:17:05.994824 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:17:17.658348 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.658313 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg"] Oct 30 17:17:17.658791 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.658614 4965 cpu_manager.go:414] "containerMap: removing container" podUID="fd74ba94-86d0-4707-94e5-6822c4ab5751" containerName="registry" Oct 30 17:17:17.658791 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.658624 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd74ba94-86d0-4707-94e5-6822c4ab5751" containerName="registry" Oct 30 17:17:17.658791 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.658637 4965 cpu_manager.go:414] "containerMap: removing container" podUID="65986964-0b39-48e4-9163-705ed74afd18" containerName="insights-gathering" Oct 30 17:17:17.658791 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.658642 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65986964-0b39-48e4-9163-705ed74afd18" containerName="insights-gathering" Oct 30 17:17:17.658791 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.658652 4965 cpu_manager.go:414] "containerMap: removing container" podUID="13444fae-8e71-41ee-8489-55b8606492fb" containerName="insights-gathering" Oct 30 17:17:17.658791 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.658657 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13444fae-8e71-41ee-8489-55b8606492fb" containerName="insights-gathering" Oct 30 17:17:17.658791 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.658667 4965 cpu_manager.go:414] "containerMap: removing container" podUID="072395f0-b9b7-438a-83d6-9f92feb714ca" containerName="console" Oct 30 17:17:17.658791 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.658672 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="072395f0-b9b7-438a-83d6-9f92feb714ca" containerName="console" Oct 30 17:17:17.658791 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.658690 4965 cpu_manager.go:414] "containerMap: removing container" podUID="b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce" containerName="registry" Oct 30 17:17:17.658791 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.658695 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce" containerName="registry" Oct 30 17:17:17.658791 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.658766 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="b69bb7ff-4506-40a7-9eb8-d2f4d4b976ce" containerName="registry" Oct 30 17:17:17.658791 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.658775 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd74ba94-86d0-4707-94e5-6822c4ab5751" containerName="registry" Oct 30 17:17:17.658791 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.658782 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="65986964-0b39-48e4-9163-705ed74afd18" containerName="insights-gathering" Oct 30 17:17:17.658791 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.658786 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="13444fae-8e71-41ee-8489-55b8606492fb" containerName="insights-gathering" Oct 30 17:17:17.658791 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.658792 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="072395f0-b9b7-438a-83d6-9f92feb714ca" containerName="console" Oct 30 17:17:17.660766 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.660751 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg" Oct 30 17:17:17.671437 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.671411 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/485b42c7-549a-479f-a4f9-62622fd84522-manifests\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg\" (UID: \"485b42c7-549a-479f-a4f9-62622fd84522\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg" Oct 30 17:17:17.671580 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.671464 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/485b42c7-549a-479f-a4f9-62622fd84522-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg\" (UID: \"485b42c7-549a-479f-a4f9-62622fd84522\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg" Oct 30 17:17:17.671580 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.671527 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/485b42c7-549a-479f-a4f9-62622fd84522-tmp\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg\" (UID: \"485b42c7-549a-479f-a4f9-62622fd84522\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg" Oct 30 17:17:17.671580 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.671570 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/485b42c7-549a-479f-a4f9-62622fd84522-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg\" (UID: \"485b42c7-549a-479f-a4f9-62622fd84522\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg" Oct 30 17:17:17.671752 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.671692 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6n8j\" (UniqueName: \"kubernetes.io/projected/485b42c7-549a-479f-a4f9-62622fd84522-kube-api-access-w6n8j\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg\" (UID: \"485b42c7-549a-479f-a4f9-62622fd84522\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg" Oct 30 17:17:17.671752 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.671742 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/485b42c7-549a-479f-a4f9-62622fd84522-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg\" (UID: \"485b42c7-549a-479f-a4f9-62622fd84522\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg" Oct 30 17:17:17.677397 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.677372 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg"] Oct 30 17:17:17.739365 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.739333 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 17:17:17.739523 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.739461 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 17:17:17.744384 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.744359 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 17:17:17.744506 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.744428 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-qkrcd\"" Oct 30 17:17:17.744506 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.744433 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 17:17:17.744572 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.744517 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 17:17:17.772992 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.772964 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/485b42c7-549a-479f-a4f9-62622fd84522-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg\" (UID: \"485b42c7-549a-479f-a4f9-62622fd84522\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg" Oct 30 17:17:17.773112 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.772993 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/485b42c7-549a-479f-a4f9-62622fd84522-tmp\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg\" (UID: \"485b42c7-549a-479f-a4f9-62622fd84522\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg" Oct 30 17:17:17.773112 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.773013 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/485b42c7-549a-479f-a4f9-62622fd84522-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg\" (UID: \"485b42c7-549a-479f-a4f9-62622fd84522\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg" Oct 30 17:17:17.773112 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.773059 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6n8j\" (UniqueName: \"kubernetes.io/projected/485b42c7-549a-479f-a4f9-62622fd84522-kube-api-access-w6n8j\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg\" (UID: \"485b42c7-549a-479f-a4f9-62622fd84522\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg" Oct 30 17:17:17.773112 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.773075 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/485b42c7-549a-479f-a4f9-62622fd84522-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg\" (UID: \"485b42c7-549a-479f-a4f9-62622fd84522\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg" Oct 30 17:17:17.773112 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.773103 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/485b42c7-549a-479f-a4f9-62622fd84522-manifests\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg\" (UID: \"485b42c7-549a-479f-a4f9-62622fd84522\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg" Oct 30 17:17:17.773518 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.773493 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/485b42c7-549a-479f-a4f9-62622fd84522-manifests\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg\" (UID: \"485b42c7-549a-479f-a4f9-62622fd84522\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg" Oct 30 17:17:17.775401 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.775377 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/485b42c7-549a-479f-a4f9-62622fd84522-tmp\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg\" (UID: \"485b42c7-549a-479f-a4f9-62622fd84522\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg" Oct 30 17:17:17.775624 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.775602 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/485b42c7-549a-479f-a4f9-62622fd84522-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg\" (UID: \"485b42c7-549a-479f-a4f9-62622fd84522\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg" Oct 30 17:17:17.775743 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.775725 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/485b42c7-549a-479f-a4f9-62622fd84522-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg\" (UID: \"485b42c7-549a-479f-a4f9-62622fd84522\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg" Oct 30 17:17:17.775801 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.775735 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/485b42c7-549a-479f-a4f9-62622fd84522-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg\" (UID: \"485b42c7-549a-479f-a4f9-62622fd84522\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg" Oct 30 17:17:17.796153 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.796126 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6n8j\" (UniqueName: \"kubernetes.io/projected/485b42c7-549a-479f-a4f9-62622fd84522-kube-api-access-w6n8j\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg\" (UID: \"485b42c7-549a-479f-a4f9-62622fd84522\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg" Oct 30 17:17:17.973348 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:17.973264 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg" Oct 30 17:17:18.329433 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:18.329407 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg"] Oct 30 17:17:18.331236 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:17:18.331209 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod485b42c7_549a_479f_a4f9_62622fd84522.slice/crio-c45ef2cb252ee5a79766c6fdf4d0a4d2f031bd0ac5473055a30944e0802593db WatchSource:0}: Error finding container c45ef2cb252ee5a79766c6fdf4d0a4d2f031bd0ac5473055a30944e0802593db: Status 404 returned error can't find the container with id c45ef2cb252ee5a79766c6fdf4d0a4d2f031bd0ac5473055a30944e0802593db Oct 30 17:17:18.452259 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:18.452221 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg" event={"ID":"485b42c7-549a-479f-a4f9-62622fd84522","Type":"ContainerStarted","Data":"c45ef2cb252ee5a79766c6fdf4d0a4d2f031bd0ac5473055a30944e0802593db"} Oct 30 17:17:21.466546 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:21.466506 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg" event={"ID":"485b42c7-549a-479f-a4f9-62622fd84522","Type":"ContainerStarted","Data":"3282457c9b5b007250f171d135759d4882bea208e44a9f47abf591b4bdc4d075"} Oct 30 17:17:22.118747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:22.118667 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 17:17:22.120074 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:22.120050 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 17:17:22.472524 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:22.472434 4965 generic.go:356] "Generic (PLEG): container finished" podID="485b42c7-549a-479f-a4f9-62622fd84522" containerID="3282457c9b5b007250f171d135759d4882bea208e44a9f47abf591b4bdc4d075" exitCode=0 Oct 30 17:17:22.472907 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:22.472528 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg" event={"ID":"485b42c7-549a-479f-a4f9-62622fd84522","Type":"ContainerDied","Data":"3282457c9b5b007250f171d135759d4882bea208e44a9f47abf591b4bdc4d075"} Oct 30 17:17:23.478589 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:23.478552 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg" event={"ID":"485b42c7-549a-479f-a4f9-62622fd84522","Type":"ContainerStarted","Data":"62423582ed96a8572fc228d299ec2502fd6967b1fb899a3d1f9cb56411ad477c"} Oct 30 17:17:23.478986 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:23.478678 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg" Oct 30 17:17:23.546457 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:23.546405 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg" podStartSLOduration=2.7433601 podStartE2EDuration="6.546392883s" totalImagesPullingTime="3.244921488s" totalInitContainerRuntime="558.111295ms" isStatefulPod=false podCreationTimestamp="2026-10-30 17:17:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:17:23.545873076 +0000 UTC m=+301.597973016" watchObservedRunningTime="2026-10-30 17:17:23.546392883 +0000 UTC m=+301.598492792" Oct 30 17:17:34.485752 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:34.485674 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg" Oct 30 17:17:35.995639 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:17:35.995602 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:17:47.524031 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:47.523999 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-57db9d4f85-w5gx5"] Oct 30 17:17:47.529330 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:47.529310 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-57db9d4f85-w5gx5" Oct 30 17:17:47.547085 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:47.547063 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-57db9d4f85-w5gx5"] Oct 30 17:17:47.557965 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:47.557931 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 17:17:47.557965 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:47.557945 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 17:17:47.558135 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:47.558026 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-j56w8\"" Oct 30 17:17:47.634488 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:47.634451 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csvtl\" (UniqueName: \"kubernetes.io/projected/463b3376-b504-4399-be15-f81c8e1952e3-kube-api-access-csvtl\") pod \"kube-auth-proxy-57db9d4f85-w5gx5\" (UID: \"463b3376-b504-4399-be15-f81c8e1952e3\") " pod="openshift-ingress/kube-auth-proxy-57db9d4f85-w5gx5" Oct 30 17:17:47.634649 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:47.634568 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/463b3376-b504-4399-be15-f81c8e1952e3-tls-certs\") pod \"kube-auth-proxy-57db9d4f85-w5gx5\" (UID: \"463b3376-b504-4399-be15-f81c8e1952e3\") " pod="openshift-ingress/kube-auth-proxy-57db9d4f85-w5gx5" Oct 30 17:17:47.634649 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:47.634608 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/463b3376-b504-4399-be15-f81c8e1952e3-tmp\") pod \"kube-auth-proxy-57db9d4f85-w5gx5\" (UID: \"463b3376-b504-4399-be15-f81c8e1952e3\") " pod="openshift-ingress/kube-auth-proxy-57db9d4f85-w5gx5" Oct 30 17:17:47.735305 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:47.735270 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/463b3376-b504-4399-be15-f81c8e1952e3-tls-certs\") pod \"kube-auth-proxy-57db9d4f85-w5gx5\" (UID: \"463b3376-b504-4399-be15-f81c8e1952e3\") " pod="openshift-ingress/kube-auth-proxy-57db9d4f85-w5gx5" Oct 30 17:17:47.735494 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:47.735313 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/463b3376-b504-4399-be15-f81c8e1952e3-tmp\") pod \"kube-auth-proxy-57db9d4f85-w5gx5\" (UID: \"463b3376-b504-4399-be15-f81c8e1952e3\") " pod="openshift-ingress/kube-auth-proxy-57db9d4f85-w5gx5" Oct 30 17:17:47.735494 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:47.735352 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csvtl\" (UniqueName: \"kubernetes.io/projected/463b3376-b504-4399-be15-f81c8e1952e3-kube-api-access-csvtl\") pod \"kube-auth-proxy-57db9d4f85-w5gx5\" (UID: \"463b3376-b504-4399-be15-f81c8e1952e3\") " pod="openshift-ingress/kube-auth-proxy-57db9d4f85-w5gx5" Oct 30 17:17:47.737546 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:47.737518 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/463b3376-b504-4399-be15-f81c8e1952e3-tmp\") pod \"kube-auth-proxy-57db9d4f85-w5gx5\" (UID: \"463b3376-b504-4399-be15-f81c8e1952e3\") " pod="openshift-ingress/kube-auth-proxy-57db9d4f85-w5gx5" Oct 30 17:17:47.737732 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:47.737713 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/463b3376-b504-4399-be15-f81c8e1952e3-tls-certs\") pod \"kube-auth-proxy-57db9d4f85-w5gx5\" (UID: \"463b3376-b504-4399-be15-f81c8e1952e3\") " pod="openshift-ingress/kube-auth-proxy-57db9d4f85-w5gx5" Oct 30 17:17:47.759064 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:47.759034 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csvtl\" (UniqueName: \"kubernetes.io/projected/463b3376-b504-4399-be15-f81c8e1952e3-kube-api-access-csvtl\") pod \"kube-auth-proxy-57db9d4f85-w5gx5\" (UID: \"463b3376-b504-4399-be15-f81c8e1952e3\") " pod="openshift-ingress/kube-auth-proxy-57db9d4f85-w5gx5" Oct 30 17:17:47.840001 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:47.839979 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-57db9d4f85-w5gx5" Oct 30 17:17:48.134078 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:48.134053 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-57db9d4f85-w5gx5"] Oct 30 17:17:48.136439 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:17:48.136415 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod463b3376_b504_4399_be15_f81c8e1952e3.slice/crio-f56cfb79ffc424f4a6db5d68811a71edcbd3bcc17d2fad4837ae5f5e062e9784 WatchSource:0}: Error finding container f56cfb79ffc424f4a6db5d68811a71edcbd3bcc17d2fad4837ae5f5e062e9784: Status 404 returned error can't find the container with id f56cfb79ffc424f4a6db5d68811a71edcbd3bcc17d2fad4837ae5f5e062e9784 Oct 30 17:17:48.138195 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:48.138175 4965 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:17:48.576746 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:48.576717 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-57db9d4f85-w5gx5" event={"ID":"463b3376-b504-4399-be15-f81c8e1952e3","Type":"ContainerStarted","Data":"f56cfb79ffc424f4a6db5d68811a71edcbd3bcc17d2fad4837ae5f5e062e9784"} Oct 30 17:17:49.132940 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:49.132905 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz"] Oct 30 17:17:49.135758 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:49.135735 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" Oct 30 17:17:49.150980 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:49.150958 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 17:17:49.162812 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:49.162723 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 17:17:49.162812 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:49.162731 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 17:17:49.162812 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:49.162791 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 17:17:49.162812 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:49.162793 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-m72mh\"" Oct 30 17:17:49.164910 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:49.164884 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz"] Oct 30 17:17:49.247753 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:49.247716 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/238e0179-5791-4664-a706-4a3d0cd3b2d0-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-8mvjz\" (UID: \"238e0179-5791-4664-a706-4a3d0cd3b2d0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" Oct 30 17:17:49.247942 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:49.247797 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/238e0179-5791-4664-a706-4a3d0cd3b2d0-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-8mvjz\" (UID: \"238e0179-5791-4664-a706-4a3d0cd3b2d0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" Oct 30 17:17:49.247942 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:49.247840 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/238e0179-5791-4664-a706-4a3d0cd3b2d0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-8mvjz\" (UID: \"238e0179-5791-4664-a706-4a3d0cd3b2d0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" Oct 30 17:17:49.247942 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:49.247881 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/238e0179-5791-4664-a706-4a3d0cd3b2d0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-8mvjz\" (UID: \"238e0179-5791-4664-a706-4a3d0cd3b2d0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" Oct 30 17:17:49.247942 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:49.247908 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22nv6\" (UniqueName: \"kubernetes.io/projected/238e0179-5791-4664-a706-4a3d0cd3b2d0-kube-api-access-22nv6\") pod \"istiod-openshift-gateway-7876bfcb74-8mvjz\" (UID: \"238e0179-5791-4664-a706-4a3d0cd3b2d0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" Oct 30 17:17:49.248152 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:49.247968 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/238e0179-5791-4664-a706-4a3d0cd3b2d0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-8mvjz\" (UID: \"238e0179-5791-4664-a706-4a3d0cd3b2d0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" Oct 30 17:17:49.248152 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:49.248000 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/238e0179-5791-4664-a706-4a3d0cd3b2d0-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-8mvjz\" (UID: \"238e0179-5791-4664-a706-4a3d0cd3b2d0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" Oct 30 17:17:49.349759 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:49.349189 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/238e0179-5791-4664-a706-4a3d0cd3b2d0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-8mvjz\" (UID: \"238e0179-5791-4664-a706-4a3d0cd3b2d0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" Oct 30 17:17:49.349759 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:49.349237 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/238e0179-5791-4664-a706-4a3d0cd3b2d0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-8mvjz\" (UID: \"238e0179-5791-4664-a706-4a3d0cd3b2d0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" Oct 30 17:17:49.349759 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:49.349264 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22nv6\" (UniqueName: \"kubernetes.io/projected/238e0179-5791-4664-a706-4a3d0cd3b2d0-kube-api-access-22nv6\") pod \"istiod-openshift-gateway-7876bfcb74-8mvjz\" (UID: \"238e0179-5791-4664-a706-4a3d0cd3b2d0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" Oct 30 17:17:49.349759 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:49.349301 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/238e0179-5791-4664-a706-4a3d0cd3b2d0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-8mvjz\" (UID: \"238e0179-5791-4664-a706-4a3d0cd3b2d0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" Oct 30 17:17:49.349759 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:49.349329 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/238e0179-5791-4664-a706-4a3d0cd3b2d0-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-8mvjz\" (UID: \"238e0179-5791-4664-a706-4a3d0cd3b2d0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" Oct 30 17:17:49.349759 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:49.349371 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/238e0179-5791-4664-a706-4a3d0cd3b2d0-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-8mvjz\" (UID: \"238e0179-5791-4664-a706-4a3d0cd3b2d0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" Oct 30 17:17:49.349759 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:49.349406 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/238e0179-5791-4664-a706-4a3d0cd3b2d0-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-8mvjz\" (UID: \"238e0179-5791-4664-a706-4a3d0cd3b2d0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" Oct 30 17:17:49.351279 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:49.351250 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/238e0179-5791-4664-a706-4a3d0cd3b2d0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-8mvjz\" (UID: \"238e0179-5791-4664-a706-4a3d0cd3b2d0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" Oct 30 17:17:49.352432 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:49.352409 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/238e0179-5791-4664-a706-4a3d0cd3b2d0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-8mvjz\" (UID: \"238e0179-5791-4664-a706-4a3d0cd3b2d0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" Oct 30 17:17:49.352699 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:49.352678 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/238e0179-5791-4664-a706-4a3d0cd3b2d0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-8mvjz\" (UID: \"238e0179-5791-4664-a706-4a3d0cd3b2d0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" Oct 30 17:17:49.353066 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:49.353033 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/238e0179-5791-4664-a706-4a3d0cd3b2d0-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-8mvjz\" (UID: \"238e0179-5791-4664-a706-4a3d0cd3b2d0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" Oct 30 17:17:49.353823 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:49.353803 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/238e0179-5791-4664-a706-4a3d0cd3b2d0-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-8mvjz\" (UID: \"238e0179-5791-4664-a706-4a3d0cd3b2d0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" Oct 30 17:17:49.373602 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:49.373580 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/238e0179-5791-4664-a706-4a3d0cd3b2d0-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-8mvjz\" (UID: \"238e0179-5791-4664-a706-4a3d0cd3b2d0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" Oct 30 17:17:49.373925 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:49.373907 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22nv6\" (UniqueName: \"kubernetes.io/projected/238e0179-5791-4664-a706-4a3d0cd3b2d0-kube-api-access-22nv6\") pod \"istiod-openshift-gateway-7876bfcb74-8mvjz\" (UID: \"238e0179-5791-4664-a706-4a3d0cd3b2d0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" Oct 30 17:17:49.449921 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:49.449520 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" Oct 30 17:17:49.771597 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:49.771558 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz"] Oct 30 17:17:49.774147 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:17:49.774108 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod238e0179_5791_4664_a706_4a3d0cd3b2d0.slice/crio-315f9cfb8fac55525c49c8eba41d831a657bc581607548047f11c7ad92f693f4 WatchSource:0}: Error finding container 315f9cfb8fac55525c49c8eba41d831a657bc581607548047f11c7ad92f693f4: Status 404 returned error can't find the container with id 315f9cfb8fac55525c49c8eba41d831a657bc581607548047f11c7ad92f693f4 Oct 30 17:17:50.589079 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:50.589045 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" event={"ID":"238e0179-5791-4664-a706-4a3d0cd3b2d0","Type":"ContainerStarted","Data":"315f9cfb8fac55525c49c8eba41d831a657bc581607548047f11c7ad92f693f4"} Oct 30 17:17:50.590812 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:50.590786 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-57db9d4f85-w5gx5" event={"ID":"463b3376-b504-4399-be15-f81c8e1952e3","Type":"ContainerStarted","Data":"e84b37d7970acf2c66d8afed79da6102ee62b936778ea243d5f2c5e62fdb464d"} Oct 30 17:17:50.632138 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:50.632077 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-57db9d4f85-w5gx5" podStartSLOduration=1.25480693 podStartE2EDuration="3.632058417s" totalImagesPullingTime="2.377251487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:17:50.63166713 +0000 UTC m=+328.683767062" watchObservedRunningTime="2026-10-30 17:17:50.632058417 +0000 UTC m=+328.684158349" Oct 30 17:17:52.347738 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:52.347705 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:17:52.348076 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:52.347771 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:17:52.601076 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:52.600992 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" event={"ID":"238e0179-5791-4664-a706-4a3d0cd3b2d0","Type":"ContainerStarted","Data":"bfe9f2b151a30f707cfa225b8a40100d98a008da56dc5bab2784c84d09c1c0b8"} Oct 30 17:17:52.601076 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:52.601059 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" Oct 30 17:17:52.669652 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:52.669591 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" podStartSLOduration=1.099370998 podStartE2EDuration="3.66957696s" totalImagesPullingTime="2.570205962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:17:52.656676969 +0000 UTC m=+330.708776900" watchObservedRunningTime="2026-10-30 17:17:52.66957696 +0000 UTC m=+330.721676891" Oct 30 17:17:53.608091 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:53.608061 4965 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-8mvjz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:17:53.608476 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:53.608105 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" podUID="238e0179-5791-4664-a706-4a3d0cd3b2d0" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:17:56.607469 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:17:56.607435 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" Oct 30 17:18:05.995316 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:18:05.995278 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:18:15.598824 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:18:15.598787 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-4b6zt"] Oct 30 17:18:15.601155 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:18:15.601134 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4b6zt" Oct 30 17:18:15.616008 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:18:15.615988 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-nhfc4\"" Oct 30 17:18:15.652800 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:18:15.652749 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 17:18:15.654229 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:18:15.654187 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-4b6zt"] Oct 30 17:18:15.665146 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:18:15.665114 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1f3cc1b1-0978-4c20-bea2-96329cc8d1c3-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-4b6zt\" (UID: \"1f3cc1b1-0978-4c20-bea2-96329cc8d1c3\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4b6zt" Oct 30 17:18:15.665291 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:18:15.665170 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49m42\" (UniqueName: \"kubernetes.io/projected/1f3cc1b1-0978-4c20-bea2-96329cc8d1c3-kube-api-access-49m42\") pod \"kserve-module-controller-manager-5cfb77bf8c-4b6zt\" (UID: \"1f3cc1b1-0978-4c20-bea2-96329cc8d1c3\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4b6zt" Oct 30 17:18:15.665361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:18:15.665305 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1f3cc1b1-0978-4c20-bea2-96329cc8d1c3-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-4b6zt\" (UID: \"1f3cc1b1-0978-4c20-bea2-96329cc8d1c3\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4b6zt" Oct 30 17:18:15.665406 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:18:15.665355 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f3cc1b1-0978-4c20-bea2-96329cc8d1c3-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-4b6zt\" (UID: \"1f3cc1b1-0978-4c20-bea2-96329cc8d1c3\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4b6zt" Oct 30 17:18:15.766525 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:18:15.766489 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f3cc1b1-0978-4c20-bea2-96329cc8d1c3-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-4b6zt\" (UID: \"1f3cc1b1-0978-4c20-bea2-96329cc8d1c3\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4b6zt" Oct 30 17:18:15.766698 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:18:15.766540 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1f3cc1b1-0978-4c20-bea2-96329cc8d1c3-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-4b6zt\" (UID: \"1f3cc1b1-0978-4c20-bea2-96329cc8d1c3\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4b6zt" Oct 30 17:18:15.766698 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:18:15.766594 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49m42\" (UniqueName: \"kubernetes.io/projected/1f3cc1b1-0978-4c20-bea2-96329cc8d1c3-kube-api-access-49m42\") pod \"kserve-module-controller-manager-5cfb77bf8c-4b6zt\" (UID: \"1f3cc1b1-0978-4c20-bea2-96329cc8d1c3\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4b6zt" Oct 30 17:18:15.766698 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:18:15.766656 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1f3cc1b1-0978-4c20-bea2-96329cc8d1c3-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-4b6zt\" (UID: \"1f3cc1b1-0978-4c20-bea2-96329cc8d1c3\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4b6zt" Oct 30 17:18:15.767009 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:18:15.766985 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f3cc1b1-0978-4c20-bea2-96329cc8d1c3-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-4b6zt\" (UID: \"1f3cc1b1-0978-4c20-bea2-96329cc8d1c3\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4b6zt" Oct 30 17:18:15.767102 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:18:15.767036 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1f3cc1b1-0978-4c20-bea2-96329cc8d1c3-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-4b6zt\" (UID: \"1f3cc1b1-0978-4c20-bea2-96329cc8d1c3\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4b6zt" Oct 30 17:18:15.767442 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:18:15.767422 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1f3cc1b1-0978-4c20-bea2-96329cc8d1c3-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-4b6zt\" (UID: \"1f3cc1b1-0978-4c20-bea2-96329cc8d1c3\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4b6zt" Oct 30 17:18:15.790330 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:18:15.790301 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49m42\" (UniqueName: \"kubernetes.io/projected/1f3cc1b1-0978-4c20-bea2-96329cc8d1c3-kube-api-access-49m42\") pod \"kserve-module-controller-manager-5cfb77bf8c-4b6zt\" (UID: \"1f3cc1b1-0978-4c20-bea2-96329cc8d1c3\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4b6zt" Oct 30 17:18:15.914262 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:18:15.914177 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4b6zt" Oct 30 17:18:16.223010 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:18:16.222982 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-4b6zt"] Oct 30 17:18:16.225479 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:18:16.225449 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3cc1b1_0978_4c20_bea2_96329cc8d1c3.slice/crio-522685540eb5c1a567868bc87e0c7d7d8af89c3e4c5f4cc8c9a5b4770b3cccb4 WatchSource:0}: Error finding container 522685540eb5c1a567868bc87e0c7d7d8af89c3e4c5f4cc8c9a5b4770b3cccb4: Status 404 returned error can't find the container with id 522685540eb5c1a567868bc87e0c7d7d8af89c3e4c5f4cc8c9a5b4770b3cccb4 Oct 30 17:18:16.696083 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:18:16.696048 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4b6zt" event={"ID":"1f3cc1b1-0978-4c20-bea2-96329cc8d1c3","Type":"ContainerStarted","Data":"522685540eb5c1a567868bc87e0c7d7d8af89c3e4c5f4cc8c9a5b4770b3cccb4"} Oct 30 17:18:18.707100 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:18:18.707063 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4b6zt" event={"ID":"1f3cc1b1-0978-4c20-bea2-96329cc8d1c3","Type":"ContainerStarted","Data":"2405a70ec3d86470ab922b8cc09524340bfd82378524ef389c67c1867f2c0a6a"} Oct 30 17:18:18.707464 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:18:18.707201 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4b6zt" Oct 30 17:18:18.763133 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:18:18.763087 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4b6zt" podStartSLOduration=2.098351362 podStartE2EDuration="3.763074897s" totalImagesPullingTime="1.664723535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:18:18.761490667 +0000 UTC m=+356.813590597" watchObservedRunningTime="2026-10-30 17:18:18.763074897 +0000 UTC m=+356.815174827" Oct 30 17:18:35.994645 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:18:35.994600 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:18:39.773148 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:18:39.773116 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-4b6zt" Oct 30 17:19:01.640071 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:01.640036 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj"] Oct 30 17:19:01.648726 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:01.648701 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj" Oct 30 17:19:01.665211 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:01.665176 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj"] Oct 30 17:19:01.673214 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:01.673184 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/284ed8f0-77ec-4f6c-8dd1-326a600ae840-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpfjj\" (UID: \"284ed8f0-77ec-4f6c-8dd1-326a600ae840\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj" Oct 30 17:19:01.673365 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:01.673223 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/284ed8f0-77ec-4f6c-8dd1-326a600ae840-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpfjj\" (UID: \"284ed8f0-77ec-4f6c-8dd1-326a600ae840\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj" Oct 30 17:19:01.673365 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:01.673287 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg9sg\" (UniqueName: \"kubernetes.io/projected/284ed8f0-77ec-4f6c-8dd1-326a600ae840-kube-api-access-qg9sg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpfjj\" (UID: \"284ed8f0-77ec-4f6c-8dd1-326a600ae840\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj" Oct 30 17:19:01.673365 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:01.673333 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/284ed8f0-77ec-4f6c-8dd1-326a600ae840-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpfjj\" (UID: \"284ed8f0-77ec-4f6c-8dd1-326a600ae840\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj" Oct 30 17:19:01.673365 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:01.673355 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/284ed8f0-77ec-4f6c-8dd1-326a600ae840-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpfjj\" (UID: \"284ed8f0-77ec-4f6c-8dd1-326a600ae840\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj" Oct 30 17:19:01.673597 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:01.673384 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/284ed8f0-77ec-4f6c-8dd1-326a600ae840-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpfjj\" (UID: \"284ed8f0-77ec-4f6c-8dd1-326a600ae840\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj" Oct 30 17:19:01.673597 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:01.673448 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/284ed8f0-77ec-4f6c-8dd1-326a600ae840-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpfjj\" (UID: \"284ed8f0-77ec-4f6c-8dd1-326a600ae840\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj" Oct 30 17:19:01.773933 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:01.773899 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/284ed8f0-77ec-4f6c-8dd1-326a600ae840-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpfjj\" (UID: \"284ed8f0-77ec-4f6c-8dd1-326a600ae840\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj" Oct 30 17:19:01.774194 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:01.773947 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/284ed8f0-77ec-4f6c-8dd1-326a600ae840-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpfjj\" (UID: \"284ed8f0-77ec-4f6c-8dd1-326a600ae840\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj" Oct 30 17:19:01.774194 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:01.773973 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/284ed8f0-77ec-4f6c-8dd1-326a600ae840-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpfjj\" (UID: \"284ed8f0-77ec-4f6c-8dd1-326a600ae840\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj" Oct 30 17:19:01.774194 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:01.773993 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/284ed8f0-77ec-4f6c-8dd1-326a600ae840-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpfjj\" (UID: \"284ed8f0-77ec-4f6c-8dd1-326a600ae840\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj" Oct 30 17:19:01.774194 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:01.774055 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/284ed8f0-77ec-4f6c-8dd1-326a600ae840-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpfjj\" (UID: \"284ed8f0-77ec-4f6c-8dd1-326a600ae840\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj" Oct 30 17:19:01.774194 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:01.774090 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/284ed8f0-77ec-4f6c-8dd1-326a600ae840-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpfjj\" (UID: \"284ed8f0-77ec-4f6c-8dd1-326a600ae840\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj" Oct 30 17:19:01.774194 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:01.774119 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg9sg\" (UniqueName: \"kubernetes.io/projected/284ed8f0-77ec-4f6c-8dd1-326a600ae840-kube-api-access-qg9sg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpfjj\" (UID: \"284ed8f0-77ec-4f6c-8dd1-326a600ae840\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj" Oct 30 17:19:01.774883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:01.774833 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/284ed8f0-77ec-4f6c-8dd1-326a600ae840-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpfjj\" (UID: \"284ed8f0-77ec-4f6c-8dd1-326a600ae840\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj" Oct 30 17:19:01.776518 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:01.776480 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/284ed8f0-77ec-4f6c-8dd1-326a600ae840-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpfjj\" (UID: \"284ed8f0-77ec-4f6c-8dd1-326a600ae840\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj" Oct 30 17:19:01.776518 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:01.776506 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/284ed8f0-77ec-4f6c-8dd1-326a600ae840-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpfjj\" (UID: \"284ed8f0-77ec-4f6c-8dd1-326a600ae840\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj" Oct 30 17:19:01.776688 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:01.776516 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/284ed8f0-77ec-4f6c-8dd1-326a600ae840-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpfjj\" (UID: \"284ed8f0-77ec-4f6c-8dd1-326a600ae840\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj" Oct 30 17:19:01.776688 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:01.776613 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/284ed8f0-77ec-4f6c-8dd1-326a600ae840-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpfjj\" (UID: \"284ed8f0-77ec-4f6c-8dd1-326a600ae840\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj" Oct 30 17:19:01.797100 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:01.797079 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg9sg\" (UniqueName: \"kubernetes.io/projected/284ed8f0-77ec-4f6c-8dd1-326a600ae840-kube-api-access-qg9sg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpfjj\" (UID: \"284ed8f0-77ec-4f6c-8dd1-326a600ae840\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj" Oct 30 17:19:01.797203 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:01.797126 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/284ed8f0-77ec-4f6c-8dd1-326a600ae840-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpfjj\" (UID: \"284ed8f0-77ec-4f6c-8dd1-326a600ae840\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj" Oct 30 17:19:01.960843 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:01.960766 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj" Oct 30 17:19:02.288594 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:02.288569 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj"] Oct 30 17:19:02.291420 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:19:02.291394 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod284ed8f0_77ec_4f6c_8dd1_326a600ae840.slice/crio-c127c4ae33e55a476952d4e27132d5d5db3192d56575488c9fe60d69e8c3d47c WatchSource:0}: Error finding container c127c4ae33e55a476952d4e27132d5d5db3192d56575488c9fe60d69e8c3d47c: Status 404 returned error can't find the container with id c127c4ae33e55a476952d4e27132d5d5db3192d56575488c9fe60d69e8c3d47c Oct 30 17:19:02.293401 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:02.293365 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:19:02.293458 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:02.293442 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:19:02.887752 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:02.887711 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj" event={"ID":"284ed8f0-77ec-4f6c-8dd1-326a600ae840","Type":"ContainerStarted","Data":"1f110e7bd6b05758a8a5f05e39a4d37c33978d510e9654d9e93f3f534b26d3f6"} Oct 30 17:19:02.888198 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:02.887769 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj" event={"ID":"284ed8f0-77ec-4f6c-8dd1-326a600ae840","Type":"ContainerStarted","Data":"c127c4ae33e55a476952d4e27132d5d5db3192d56575488c9fe60d69e8c3d47c"} Oct 30 17:19:02.888198 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:02.887811 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj" Oct 30 17:19:02.950984 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:02.950882 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj" podStartSLOduration=1.950853306 podStartE2EDuration="1.950853306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:19:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:19:02.949810933 +0000 UTC m=+401.001910883" watchObservedRunningTime="2026-10-30 17:19:02.950853306 +0000 UTC m=+401.002953238" Oct 30 17:19:03.894389 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:03.894350 4965 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kpfjj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:19:03.894793 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:03.894411 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj" podUID="284ed8f0-77ec-4f6c-8dd1-326a600ae840" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:19:05.995878 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:19:05.995832 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:19:06.894004 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:06.893970 4965 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kpfjj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:19:06.894183 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:06.894029 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj" podUID="284ed8f0-77ec-4f6c-8dd1-326a600ae840" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:19:09.893164 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:09.893124 4965 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kpfjj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:19:09.893638 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:09.893187 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj" podUID="284ed8f0-77ec-4f6c-8dd1-326a600ae840" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:19:12.894076 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:12.894034 4965 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kpfjj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:19:12.894476 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:12.894109 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj" podUID="284ed8f0-77ec-4f6c-8dd1-326a600ae840" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:19:15.893366 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:15.893330 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpfjj" Oct 30 17:19:15.984613 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:15.984570 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz"] Oct 30 17:19:15.984840 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:15.984815 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" podUID="238e0179-5791-4664-a706-4a3d0cd3b2d0" containerName="discovery" containerID="cri-o://bfe9f2b151a30f707cfa225b8a40100d98a008da56dc5bab2784c84d09c1c0b8" gracePeriod=30 Oct 30 17:19:16.222604 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:16.222582 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" Oct 30 17:19:16.300455 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:16.300423 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/238e0179-5791-4664-a706-4a3d0cd3b2d0-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/238e0179-5791-4664-a706-4a3d0cd3b2d0-istio-csr-dns-cert\") pod \"238e0179-5791-4664-a706-4a3d0cd3b2d0\" (UID: \"238e0179-5791-4664-a706-4a3d0cd3b2d0\") " Oct 30 17:19:16.300618 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:16.300469 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/238e0179-5791-4664-a706-4a3d0cd3b2d0-cacerts\" (UniqueName: \"kubernetes.io/secret/238e0179-5791-4664-a706-4a3d0cd3b2d0-cacerts\") pod \"238e0179-5791-4664-a706-4a3d0cd3b2d0\" (UID: \"238e0179-5791-4664-a706-4a3d0cd3b2d0\") " Oct 30 17:19:16.300618 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:16.300491 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/238e0179-5791-4664-a706-4a3d0cd3b2d0-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/238e0179-5791-4664-a706-4a3d0cd3b2d0-istio-csr-ca-configmap\") pod \"238e0179-5791-4664-a706-4a3d0cd3b2d0\" (UID: \"238e0179-5791-4664-a706-4a3d0cd3b2d0\") " Oct 30 17:19:16.300618 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:16.300537 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/238e0179-5791-4664-a706-4a3d0cd3b2d0-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/238e0179-5791-4664-a706-4a3d0cd3b2d0-istio-kubeconfig\") pod \"238e0179-5791-4664-a706-4a3d0cd3b2d0\" (UID: \"238e0179-5791-4664-a706-4a3d0cd3b2d0\") " Oct 30 17:19:16.300618 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:16.300567 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/238e0179-5791-4664-a706-4a3d0cd3b2d0-kube-api-access-22nv6\" (UniqueName: \"kubernetes.io/projected/238e0179-5791-4664-a706-4a3d0cd3b2d0-kube-api-access-22nv6\") pod \"238e0179-5791-4664-a706-4a3d0cd3b2d0\" (UID: \"238e0179-5791-4664-a706-4a3d0cd3b2d0\") " Oct 30 17:19:16.300618 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:16.300601 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/238e0179-5791-4664-a706-4a3d0cd3b2d0-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/238e0179-5791-4664-a706-4a3d0cd3b2d0-local-certs\") pod \"238e0179-5791-4664-a706-4a3d0cd3b2d0\" (UID: \"238e0179-5791-4664-a706-4a3d0cd3b2d0\") " Oct 30 17:19:16.300905 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:16.300632 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/238e0179-5791-4664-a706-4a3d0cd3b2d0-istio-token\" (UniqueName: \"kubernetes.io/projected/238e0179-5791-4664-a706-4a3d0cd3b2d0-istio-token\") pod \"238e0179-5791-4664-a706-4a3d0cd3b2d0\" (UID: \"238e0179-5791-4664-a706-4a3d0cd3b2d0\") " Oct 30 17:19:16.301071 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:16.301018 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/238e0179-5791-4664-a706-4a3d0cd3b2d0-istio-csr-ca-configmap" pod "238e0179-5791-4664-a706-4a3d0cd3b2d0" (UID: "238e0179-5791-4664-a706-4a3d0cd3b2d0"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:19:16.302996 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:16.302965 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/238e0179-5791-4664-a706-4a3d0cd3b2d0-kube-api-access-22nv6" pod "238e0179-5791-4664-a706-4a3d0cd3b2d0" (UID: "238e0179-5791-4664-a706-4a3d0cd3b2d0"). InnerVolumeSpecName "kube-api-access-22nv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:19:16.303114 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:16.303003 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/238e0179-5791-4664-a706-4a3d0cd3b2d0-istio-token" pod "238e0179-5791-4664-a706-4a3d0cd3b2d0" (UID: "238e0179-5791-4664-a706-4a3d0cd3b2d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:19:16.303114 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:16.303090 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/238e0179-5791-4664-a706-4a3d0cd3b2d0-istio-kubeconfig" pod "238e0179-5791-4664-a706-4a3d0cd3b2d0" (UID: "238e0179-5791-4664-a706-4a3d0cd3b2d0"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:19:16.303114 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:16.303106 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/238e0179-5791-4664-a706-4a3d0cd3b2d0-istio-csr-dns-cert" pod "238e0179-5791-4664-a706-4a3d0cd3b2d0" (UID: "238e0179-5791-4664-a706-4a3d0cd3b2d0"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:19:16.303234 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:16.303123 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/238e0179-5791-4664-a706-4a3d0cd3b2d0-cacerts" pod "238e0179-5791-4664-a706-4a3d0cd3b2d0" (UID: "238e0179-5791-4664-a706-4a3d0cd3b2d0"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:19:16.303398 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:16.303368 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/238e0179-5791-4664-a706-4a3d0cd3b2d0-local-certs" pod "238e0179-5791-4664-a706-4a3d0cd3b2d0" (UID: "238e0179-5791-4664-a706-4a3d0cd3b2d0"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:19:16.401580 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:16.401502 4965 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/238e0179-5791-4664-a706-4a3d0cd3b2d0-istio-kubeconfig\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:19:16.401580 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:16.401535 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22nv6\" (UniqueName: \"kubernetes.io/projected/238e0179-5791-4664-a706-4a3d0cd3b2d0-kube-api-access-22nv6\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:19:16.401580 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:16.401546 4965 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/238e0179-5791-4664-a706-4a3d0cd3b2d0-local-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:19:16.401580 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:16.401554 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/238e0179-5791-4664-a706-4a3d0cd3b2d0-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:19:16.401580 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:16.401563 4965 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/238e0179-5791-4664-a706-4a3d0cd3b2d0-istio-csr-dns-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:19:16.401580 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:16.401572 4965 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/238e0179-5791-4664-a706-4a3d0cd3b2d0-cacerts\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:19:16.401580 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:16.401582 4965 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/238e0179-5791-4664-a706-4a3d0cd3b2d0-istio-csr-ca-configmap\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:19:16.946484 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:16.946445 4965 generic.go:356] "Generic (PLEG): container finished" podID="238e0179-5791-4664-a706-4a3d0cd3b2d0" containerID="bfe9f2b151a30f707cfa225b8a40100d98a008da56dc5bab2784c84d09c1c0b8" exitCode=0 Oct 30 17:19:16.946914 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:16.946502 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" Oct 30 17:19:16.946914 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:16.946517 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" event={"ID":"238e0179-5791-4664-a706-4a3d0cd3b2d0","Type":"ContainerDied","Data":"bfe9f2b151a30f707cfa225b8a40100d98a008da56dc5bab2784c84d09c1c0b8"} Oct 30 17:19:16.946914 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:16.946591 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz" event={"ID":"238e0179-5791-4664-a706-4a3d0cd3b2d0","Type":"ContainerDied","Data":"315f9cfb8fac55525c49c8eba41d831a657bc581607548047f11c7ad92f693f4"} Oct 30 17:19:16.946914 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:16.946613 4965 scope.go:122] "RemoveContainer" containerID="bfe9f2b151a30f707cfa225b8a40100d98a008da56dc5bab2784c84d09c1c0b8" Oct 30 17:19:16.955663 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:16.955647 4965 scope.go:122] "RemoveContainer" containerID="bfe9f2b151a30f707cfa225b8a40100d98a008da56dc5bab2784c84d09c1c0b8" Oct 30 17:19:16.955922 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:19:16.955902 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfe9f2b151a30f707cfa225b8a40100d98a008da56dc5bab2784c84d09c1c0b8\": container with ID starting with bfe9f2b151a30f707cfa225b8a40100d98a008da56dc5bab2784c84d09c1c0b8 not found: ID does not exist" containerID="bfe9f2b151a30f707cfa225b8a40100d98a008da56dc5bab2784c84d09c1c0b8" Oct 30 17:19:16.955969 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:16.955930 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfe9f2b151a30f707cfa225b8a40100d98a008da56dc5bab2784c84d09c1c0b8"} err="failed to get container status \"bfe9f2b151a30f707cfa225b8a40100d98a008da56dc5bab2784c84d09c1c0b8\": rpc error: code = NotFound desc = could not find container \"bfe9f2b151a30f707cfa225b8a40100d98a008da56dc5bab2784c84d09c1c0b8\": container with ID starting with bfe9f2b151a30f707cfa225b8a40100d98a008da56dc5bab2784c84d09c1c0b8 not found: ID does not exist" Oct 30 17:19:17.009766 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:17.009738 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz"] Oct 30 17:19:17.022322 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:17.022290 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8mvjz"] Oct 30 17:19:18.096191 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:19:18.096157 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="238e0179-5791-4664-a706-4a3d0cd3b2d0" path="/var/lib/kubelet/pods/238e0179-5791-4664-a706-4a3d0cd3b2d0/volumes" Oct 30 17:19:35.994772 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:19:35.994731 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:20:04.721848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:04.721813 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-m275c"] Oct 30 17:20:04.722342 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:04.722206 4965 cpu_manager.go:414] "containerMap: removing container" podUID="238e0179-5791-4664-a706-4a3d0cd3b2d0" containerName="discovery" Oct 30 17:20:04.722342 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:04.722218 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="238e0179-5791-4664-a706-4a3d0cd3b2d0" containerName="discovery" Oct 30 17:20:04.722342 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:04.722285 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="238e0179-5791-4664-a706-4a3d0cd3b2d0" containerName="discovery" Oct 30 17:20:04.727961 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:04.725902 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-m275c" Oct 30 17:20:04.748104 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:04.748072 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 17:20:04.748269 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:04.748137 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 17:20:04.760361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:04.760338 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-ln6nv\"" Oct 30 17:20:04.762106 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:04.762085 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-m275c"] Oct 30 17:20:04.800121 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:04.800087 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfcql\" (UniqueName: \"kubernetes.io/projected/62ded67d-b19f-4857-8852-f6b1f02c0323-kube-api-access-kfcql\") pod \"authorino-operator-68d6dd76dc-m275c\" (UID: \"62ded67d-b19f-4857-8852-f6b1f02c0323\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-m275c" Oct 30 17:20:04.901428 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:04.901394 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfcql\" (UniqueName: \"kubernetes.io/projected/62ded67d-b19f-4857-8852-f6b1f02c0323-kube-api-access-kfcql\") pod \"authorino-operator-68d6dd76dc-m275c\" (UID: \"62ded67d-b19f-4857-8852-f6b1f02c0323\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-m275c" Oct 30 17:20:04.926682 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:04.926653 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfcql\" (UniqueName: \"kubernetes.io/projected/62ded67d-b19f-4857-8852-f6b1f02c0323-kube-api-access-kfcql\") pod \"authorino-operator-68d6dd76dc-m275c\" (UID: \"62ded67d-b19f-4857-8852-f6b1f02c0323\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-m275c" Oct 30 17:20:05.039842 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:05.039755 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-m275c" Oct 30 17:20:05.340903 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:05.340876 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-m275c"] Oct 30 17:20:05.342725 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:20:05.342698 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ded67d_b19f_4857_8852_f6b1f02c0323.slice/crio-aa3348dfb58527b6d4e638ba366395e97b00bb809fdd8fc48280174e29e7b82a WatchSource:0}: Error finding container aa3348dfb58527b6d4e638ba366395e97b00bb809fdd8fc48280174e29e7b82a: Status 404 returned error can't find the container with id aa3348dfb58527b6d4e638ba366395e97b00bb809fdd8fc48280174e29e7b82a Oct 30 17:20:05.995249 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:20:05.995214 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:20:06.160217 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:06.160183 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-m275c" event={"ID":"62ded67d-b19f-4857-8852-f6b1f02c0323","Type":"ContainerStarted","Data":"aa3348dfb58527b6d4e638ba366395e97b00bb809fdd8fc48280174e29e7b82a"} Oct 30 17:20:08.170538 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:08.170446 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-m275c" event={"ID":"62ded67d-b19f-4857-8852-f6b1f02c0323","Type":"ContainerStarted","Data":"6cacde1955679a9136cfd46f745e796c8eb1e46b0329e20b8d0d8689467c6819"} Oct 30 17:20:08.170538 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:08.170515 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-m275c" Oct 30 17:20:08.228492 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:08.228442 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d6dd76dc-m275c" podStartSLOduration=1.654315504 podStartE2EDuration="4.228426729s" totalImagesPullingTime="2.574111225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:20:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:20:08.226594707 +0000 UTC m=+466.278694651" watchObservedRunningTime="2026-10-30 17:20:08.228426729 +0000 UTC m=+466.280526660" Oct 30 17:20:11.462463 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:11.462428 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-bg6nh"] Oct 30 17:20:11.465899 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:11.465882 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-bg6nh" Oct 30 17:20:11.478580 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:11.478557 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-bg6nh"] Oct 30 17:20:11.481236 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:11.481217 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4ptvx\"" Oct 30 17:20:11.561158 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:11.561117 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-bg6nh\" (UID: \"48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-bg6nh" Oct 30 17:20:11.561341 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:11.561239 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zt48\" (UniqueName: \"kubernetes.io/projected/48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2-kube-api-access-8zt48\") pod \"kuadrant-operator-controller-manager-6b6fb56846-bg6nh\" (UID: \"48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-bg6nh" Oct 30 17:20:11.662509 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:11.662453 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-bg6nh\" (UID: \"48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-bg6nh" Oct 30 17:20:11.662710 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:11.662566 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zt48\" (UniqueName: \"kubernetes.io/projected/48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2-kube-api-access-8zt48\") pod \"kuadrant-operator-controller-manager-6b6fb56846-bg6nh\" (UID: \"48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-bg6nh" Oct 30 17:20:11.662900 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:11.662849 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-bg6nh\" (UID: \"48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-bg6nh" Oct 30 17:20:11.685841 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:11.685816 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zt48\" (UniqueName: \"kubernetes.io/projected/48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2-kube-api-access-8zt48\") pod \"kuadrant-operator-controller-manager-6b6fb56846-bg6nh\" (UID: \"48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-bg6nh" Oct 30 17:20:11.778761 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:11.778687 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-bg6nh" Oct 30 17:20:12.114596 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:12.114568 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-bg6nh"] Oct 30 17:20:12.117178 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:20:12.117152 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ec95e4_661a_43ac_a6e6_18f0b9cf2ef2.slice/crio-3cfdfa57b3f093f187111c6e6f5128440cd8f6789d133afac259fc2999db8cab WatchSource:0}: Error finding container 3cfdfa57b3f093f187111c6e6f5128440cd8f6789d133afac259fc2999db8cab: Status 404 returned error can't find the container with id 3cfdfa57b3f093f187111c6e6f5128440cd8f6789d133afac259fc2999db8cab Oct 30 17:20:12.187945 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:12.187907 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-bg6nh" event={"ID":"48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2","Type":"ContainerStarted","Data":"3cfdfa57b3f093f187111c6e6f5128440cd8f6789d133afac259fc2999db8cab"} Oct 30 17:20:16.210905 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:16.210810 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-bg6nh" event={"ID":"48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2","Type":"ContainerStarted","Data":"06d494a3a4154f61f22e5b1d86045352ffe30217aeab6d0514711232199e569d"} Oct 30 17:20:16.210905 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:16.210902 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-bg6nh" Oct 30 17:20:16.279692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:16.279640 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-bg6nh" podStartSLOduration=1.502661877 podStartE2EDuration="5.279626793s" totalImagesPullingTime="3.776964916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:20:16.266362603 +0000 UTC m=+474.318462554" watchObservedRunningTime="2026-10-30 17:20:16.279626793 +0000 UTC m=+474.331726724" Oct 30 17:20:19.178310 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:19.178279 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-m275c" Oct 30 17:20:27.217812 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:27.217780 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-bg6nh" Oct 30 17:20:35.995654 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:20:35.995579 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:20:41.347099 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.347065 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-bg6nh"] Oct 30 17:20:41.347503 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.347299 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-bg6nh" podUID="48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2" containerName="manager" containerID="cri-o://06d494a3a4154f61f22e5b1d86045352ffe30217aeab6d0514711232199e569d" gracePeriod=2 Oct 30 17:20:41.353285 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.353256 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-bg6nh"] Oct 30 17:20:41.367736 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:20:41.361121 4965 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-bg6nh\" is forbidden: User \"system:node:ip-10-0-139-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-222.ec2.internal' and this object" podUID="48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-bg6nh" Oct 30 17:20:41.367736 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.365132 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kwzth"] Oct 30 17:20:41.367736 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.365476 4965 cpu_manager.go:414] "containerMap: removing container" podUID="48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2" containerName="manager" Oct 30 17:20:41.367736 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.365488 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2" containerName="manager" Oct 30 17:20:41.367736 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.365580 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2" containerName="manager" Oct 30 17:20:41.369514 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.369060 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kwzth" Oct 30 17:20:41.382946 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:20:41.382918 4965 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-bg6nh\" is forbidden: User \"system:node:ip-10-0-139-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-222.ec2.internal' and this object" podUID="48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-bg6nh" Oct 30 17:20:41.393745 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.393720 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kwzth"] Oct 30 17:20:41.408929 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.408905 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-v8fjn"] Oct 30 17:20:41.413241 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.413218 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-v8fjn" Oct 30 17:20:41.444050 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.444020 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-v8fjn"] Oct 30 17:20:41.499935 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:20:41.499883 4965 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-bg6nh\" is forbidden: User \"system:node:ip-10-0-139-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-222.ec2.internal' and this object" podUID="48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-bg6nh" Oct 30 17:20:41.519360 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.519326 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ebae7e5-7c84-41df-882c-a130a210496f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-kwzth\" (UID: \"6ebae7e5-7c84-41df-882c-a130a210496f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kwzth" Oct 30 17:20:41.519360 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.519363 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qswfb\" (UniqueName: \"kubernetes.io/projected/6ebae7e5-7c84-41df-882c-a130a210496f-kube-api-access-qswfb\") pod \"kuadrant-operator-controller-manager-6b6fb56846-kwzth\" (UID: \"6ebae7e5-7c84-41df-882c-a130a210496f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kwzth" Oct 30 17:20:41.519550 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.519527 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1b2b72a-588c-4847-8963-5f806a57964a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-v8fjn\" (UID: \"e1b2b72a-588c-4847-8963-5f806a57964a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-v8fjn" Oct 30 17:20:41.519588 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.519565 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s42wn\" (UniqueName: \"kubernetes.io/projected/e1b2b72a-588c-4847-8963-5f806a57964a-kube-api-access-s42wn\") pod \"kuadrant-operator-controller-manager-6b76784d48-v8fjn\" (UID: \"e1b2b72a-588c-4847-8963-5f806a57964a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-v8fjn" Oct 30 17:20:41.620943 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.620857 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s42wn\" (UniqueName: \"kubernetes.io/projected/e1b2b72a-588c-4847-8963-5f806a57964a-kube-api-access-s42wn\") pod \"kuadrant-operator-controller-manager-6b76784d48-v8fjn\" (UID: \"e1b2b72a-588c-4847-8963-5f806a57964a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-v8fjn" Oct 30 17:20:41.621065 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.620958 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ebae7e5-7c84-41df-882c-a130a210496f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-kwzth\" (UID: \"6ebae7e5-7c84-41df-882c-a130a210496f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kwzth" Oct 30 17:20:41.621065 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.620982 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qswfb\" (UniqueName: \"kubernetes.io/projected/6ebae7e5-7c84-41df-882c-a130a210496f-kube-api-access-qswfb\") pod \"kuadrant-operator-controller-manager-6b6fb56846-kwzth\" (UID: \"6ebae7e5-7c84-41df-882c-a130a210496f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kwzth" Oct 30 17:20:41.621065 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.621049 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1b2b72a-588c-4847-8963-5f806a57964a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-v8fjn\" (UID: \"e1b2b72a-588c-4847-8963-5f806a57964a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-v8fjn" Oct 30 17:20:41.621338 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.621312 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ebae7e5-7c84-41df-882c-a130a210496f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-kwzth\" (UID: \"6ebae7e5-7c84-41df-882c-a130a210496f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kwzth" Oct 30 17:20:41.621338 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.621334 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1b2b72a-588c-4847-8963-5f806a57964a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-v8fjn\" (UID: \"e1b2b72a-588c-4847-8963-5f806a57964a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-v8fjn" Oct 30 17:20:41.622317 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.622299 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-bg6nh" Oct 30 17:20:41.635801 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:20:41.635768 4965 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-bg6nh\" is forbidden: User \"system:node:ip-10-0-139-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-222.ec2.internal' and this object" podUID="48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-bg6nh" Oct 30 17:20:41.646920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.646889 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s42wn\" (UniqueName: \"kubernetes.io/projected/e1b2b72a-588c-4847-8963-5f806a57964a-kube-api-access-s42wn\") pod \"kuadrant-operator-controller-manager-6b76784d48-v8fjn\" (UID: \"e1b2b72a-588c-4847-8963-5f806a57964a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-v8fjn" Oct 30 17:20:41.647027 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.646928 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qswfb\" (UniqueName: \"kubernetes.io/projected/6ebae7e5-7c84-41df-882c-a130a210496f-kube-api-access-qswfb\") pod \"kuadrant-operator-controller-manager-6b6fb56846-kwzth\" (UID: \"6ebae7e5-7c84-41df-882c-a130a210496f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kwzth" Oct 30 17:20:41.721968 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.721939 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2-kube-api-access-8zt48\" (UniqueName: \"kubernetes.io/projected/48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2-kube-api-access-8zt48\") pod \"48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2\" (UID: \"48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2\") " Oct 30 17:20:41.722106 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.721976 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2-extensions-socket-volume\") pod \"48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2\" (UID: \"48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2\") " Oct 30 17:20:41.722365 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.722340 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2-extensions-socket-volume" pod "48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2" (UID: "48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:20:41.724148 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.724126 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2-kube-api-access-8zt48" pod "48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2" (UID: "48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2"). InnerVolumeSpecName "kube-api-access-8zt48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:20:41.726282 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.726265 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kwzth" Oct 30 17:20:41.736407 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.736388 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-v8fjn" Oct 30 17:20:41.823471 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.823445 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zt48\" (UniqueName: \"kubernetes.io/projected/48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2-kube-api-access-8zt48\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:20:41.823471 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:41.823472 4965 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2-extensions-socket-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:20:42.086674 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:42.086650 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kwzth"] Oct 30 17:20:42.088790 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:20:42.088766 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebae7e5_7c84_41df_882c_a130a210496f.slice/crio-bf88bede02a661f74b8ae01a45fd07ba3099530679901206ad23e330e1f7145e WatchSource:0}: Error finding container bf88bede02a661f74b8ae01a45fd07ba3099530679901206ad23e330e1f7145e: Status 404 returned error can't find the container with id bf88bede02a661f74b8ae01a45fd07ba3099530679901206ad23e330e1f7145e Oct 30 17:20:42.097022 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:42.096996 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2" path="/var/lib/kubelet/pods/48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2/volumes" Oct 30 17:20:42.097577 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:42.097559 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-v8fjn"] Oct 30 17:20:42.101711 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:20:42.101691 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1b2b72a_588c_4847_8963_5f806a57964a.slice/crio-3a8011ca2e51bb0b5ca20929c8ee05eea63cfd3a8d46f7fbada6f6d41ef14b2b WatchSource:0}: Error finding container 3a8011ca2e51bb0b5ca20929c8ee05eea63cfd3a8d46f7fbada6f6d41ef14b2b: Status 404 returned error can't find the container with id 3a8011ca2e51bb0b5ca20929c8ee05eea63cfd3a8d46f7fbada6f6d41ef14b2b Oct 30 17:20:42.106766 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:20:42.106726 4965 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-bg6nh\" is forbidden: User \"system:node:ip-10-0-139-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-222.ec2.internal' and this object" podUID="48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-bg6nh" Oct 30 17:20:42.331475 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:42.331432 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kwzth" event={"ID":"6ebae7e5-7c84-41df-882c-a130a210496f","Type":"ContainerStarted","Data":"c13b66438270caed76ee207a23d1456855468a694c3b2c8850a8acd01fad4928"} Oct 30 17:20:42.331649 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:42.331530 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kwzth" event={"ID":"6ebae7e5-7c84-41df-882c-a130a210496f","Type":"ContainerStarted","Data":"bf88bede02a661f74b8ae01a45fd07ba3099530679901206ad23e330e1f7145e"} Oct 30 17:20:42.331649 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:42.331558 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kwzth" Oct 30 17:20:42.332828 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:42.332802 4965 generic.go:356] "Generic (PLEG): container finished" podID="48ec95e4-661a-43ac-a6e6-18f0b9cf2ef2" containerID="06d494a3a4154f61f22e5b1d86045352ffe30217aeab6d0514711232199e569d" exitCode=0 Oct 30 17:20:42.332975 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:42.332885 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-bg6nh" Oct 30 17:20:42.332975 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:42.332919 4965 scope.go:122] "RemoveContainer" containerID="06d494a3a4154f61f22e5b1d86045352ffe30217aeab6d0514711232199e569d" Oct 30 17:20:42.334722 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:42.334684 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-v8fjn" event={"ID":"e1b2b72a-588c-4847-8963-5f806a57964a","Type":"ContainerStarted","Data":"16329df6da7256b6fa84b2eb74e8e63b25b3adb8ed04aa974a8c016979f3f1e8"} Oct 30 17:20:42.334820 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:42.334735 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-v8fjn" event={"ID":"e1b2b72a-588c-4847-8963-5f806a57964a","Type":"ContainerStarted","Data":"3a8011ca2e51bb0b5ca20929c8ee05eea63cfd3a8d46f7fbada6f6d41ef14b2b"} Oct 30 17:20:42.334820 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:42.334789 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-v8fjn" Oct 30 17:20:42.341323 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:42.341308 4965 scope.go:122] "RemoveContainer" containerID="06d494a3a4154f61f22e5b1d86045352ffe30217aeab6d0514711232199e569d" Oct 30 17:20:42.341576 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:20:42.341557 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06d494a3a4154f61f22e5b1d86045352ffe30217aeab6d0514711232199e569d\": container with ID starting with 06d494a3a4154f61f22e5b1d86045352ffe30217aeab6d0514711232199e569d not found: ID does not exist" containerID="06d494a3a4154f61f22e5b1d86045352ffe30217aeab6d0514711232199e569d" Oct 30 17:20:42.341633 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:42.341584 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06d494a3a4154f61f22e5b1d86045352ffe30217aeab6d0514711232199e569d"} err="failed to get container status \"06d494a3a4154f61f22e5b1d86045352ffe30217aeab6d0514711232199e569d\": rpc error: code = NotFound desc = could not find container \"06d494a3a4154f61f22e5b1d86045352ffe30217aeab6d0514711232199e569d\": container with ID starting with 06d494a3a4154f61f22e5b1d86045352ffe30217aeab6d0514711232199e569d not found: ID does not exist" Oct 30 17:20:42.377576 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:42.377527 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kwzth" podStartSLOduration=1.377510826 podStartE2EDuration="1.377510826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:20:42.37576196 +0000 UTC m=+500.427861911" watchObservedRunningTime="2026-10-30 17:20:42.377510826 +0000 UTC m=+500.429610763" Oct 30 17:20:42.423259 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:42.423216 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-v8fjn" podStartSLOduration=1.423204008 podStartE2EDuration="1.423204008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:20:42.422164796 +0000 UTC m=+500.474264726" watchObservedRunningTime="2026-10-30 17:20:42.423204008 +0000 UTC m=+500.475303939" Oct 30 17:20:53.341712 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:53.341679 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kwzth" Oct 30 17:20:53.342150 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:53.341759 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-v8fjn" Oct 30 17:20:53.496380 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:53.496344 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kwzth"] Oct 30 17:20:53.496556 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:53.496534 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kwzth" podUID="6ebae7e5-7c84-41df-882c-a130a210496f" containerName="manager" containerID="cri-o://c13b66438270caed76ee207a23d1456855468a694c3b2c8850a8acd01fad4928" gracePeriod=10 Oct 30 17:20:53.740665 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:53.740635 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kwzth" Oct 30 17:20:53.827685 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:53.827655 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ebae7e5-7c84-41df-882c-a130a210496f-kube-api-access-qswfb\" (UniqueName: \"kubernetes.io/projected/6ebae7e5-7c84-41df-882c-a130a210496f-kube-api-access-qswfb\") pod \"6ebae7e5-7c84-41df-882c-a130a210496f\" (UID: \"6ebae7e5-7c84-41df-882c-a130a210496f\") " Oct 30 17:20:53.827849 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:53.827727 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ebae7e5-7c84-41df-882c-a130a210496f-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ebae7e5-7c84-41df-882c-a130a210496f-extensions-socket-volume\") pod \"6ebae7e5-7c84-41df-882c-a130a210496f\" (UID: \"6ebae7e5-7c84-41df-882c-a130a210496f\") " Oct 30 17:20:53.828186 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:53.828145 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ebae7e5-7c84-41df-882c-a130a210496f-extensions-socket-volume" pod "6ebae7e5-7c84-41df-882c-a130a210496f" (UID: "6ebae7e5-7c84-41df-882c-a130a210496f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:20:53.829763 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:53.829735 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ebae7e5-7c84-41df-882c-a130a210496f-kube-api-access-qswfb" pod "6ebae7e5-7c84-41df-882c-a130a210496f" (UID: "6ebae7e5-7c84-41df-882c-a130a210496f"). InnerVolumeSpecName "kube-api-access-qswfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:20:53.929096 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:53.929024 4965 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ebae7e5-7c84-41df-882c-a130a210496f-extensions-socket-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:20:53.929096 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:53.929050 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qswfb\" (UniqueName: \"kubernetes.io/projected/6ebae7e5-7c84-41df-882c-a130a210496f-kube-api-access-qswfb\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:20:54.392045 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:54.392008 4965 generic.go:356] "Generic (PLEG): container finished" podID="6ebae7e5-7c84-41df-882c-a130a210496f" containerID="c13b66438270caed76ee207a23d1456855468a694c3b2c8850a8acd01fad4928" exitCode=0 Oct 30 17:20:54.392440 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:54.392072 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kwzth" Oct 30 17:20:54.392440 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:54.392101 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kwzth" event={"ID":"6ebae7e5-7c84-41df-882c-a130a210496f","Type":"ContainerDied","Data":"c13b66438270caed76ee207a23d1456855468a694c3b2c8850a8acd01fad4928"} Oct 30 17:20:54.392440 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:54.392155 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kwzth" event={"ID":"6ebae7e5-7c84-41df-882c-a130a210496f","Type":"ContainerDied","Data":"bf88bede02a661f74b8ae01a45fd07ba3099530679901206ad23e330e1f7145e"} Oct 30 17:20:54.392440 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:54.392178 4965 scope.go:122] "RemoveContainer" containerID="c13b66438270caed76ee207a23d1456855468a694c3b2c8850a8acd01fad4928" Oct 30 17:20:54.400255 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:54.400235 4965 scope.go:122] "RemoveContainer" containerID="c13b66438270caed76ee207a23d1456855468a694c3b2c8850a8acd01fad4928" Oct 30 17:20:54.400528 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:20:54.400509 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c13b66438270caed76ee207a23d1456855468a694c3b2c8850a8acd01fad4928\": container with ID starting with c13b66438270caed76ee207a23d1456855468a694c3b2c8850a8acd01fad4928 not found: ID does not exist" containerID="c13b66438270caed76ee207a23d1456855468a694c3b2c8850a8acd01fad4928" Oct 30 17:20:54.400590 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:54.400540 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c13b66438270caed76ee207a23d1456855468a694c3b2c8850a8acd01fad4928"} err="failed to get container status \"c13b66438270caed76ee207a23d1456855468a694c3b2c8850a8acd01fad4928\": rpc error: code = NotFound desc = could not find container \"c13b66438270caed76ee207a23d1456855468a694c3b2c8850a8acd01fad4928\": container with ID starting with c13b66438270caed76ee207a23d1456855468a694c3b2c8850a8acd01fad4928 not found: ID does not exist" Oct 30 17:20:54.451763 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:54.451731 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kwzth"] Oct 30 17:20:54.465397 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:54.465368 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-kwzth"] Oct 30 17:20:56.096652 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:20:56.096609 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ebae7e5-7c84-41df-882c-a130a210496f" path="/var/lib/kubelet/pods/6ebae7e5-7c84-41df-882c-a130a210496f/volumes" Oct 30 17:21:05.995795 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:21:05.995760 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:21:06.131314 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:06.131280 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-v8fjn"] Oct 30 17:21:06.131520 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:06.131478 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-v8fjn" podUID="e1b2b72a-588c-4847-8963-5f806a57964a" containerName="manager" containerID="cri-o://16329df6da7256b6fa84b2eb74e8e63b25b3adb8ed04aa974a8c016979f3f1e8" gracePeriod=10 Oct 30 17:21:06.389547 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:06.389524 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-v8fjn" Oct 30 17:21:06.430297 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:06.430271 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1b2b72a-588c-4847-8963-5f806a57964a-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1b2b72a-588c-4847-8963-5f806a57964a-extensions-socket-volume\") pod \"e1b2b72a-588c-4847-8963-5f806a57964a\" (UID: \"e1b2b72a-588c-4847-8963-5f806a57964a\") " Oct 30 17:21:06.430405 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:06.430389 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1b2b72a-588c-4847-8963-5f806a57964a-kube-api-access-s42wn\" (UniqueName: \"kubernetes.io/projected/e1b2b72a-588c-4847-8963-5f806a57964a-kube-api-access-s42wn\") pod \"e1b2b72a-588c-4847-8963-5f806a57964a\" (UID: \"e1b2b72a-588c-4847-8963-5f806a57964a\") " Oct 30 17:21:06.430693 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:06.430670 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1b2b72a-588c-4847-8963-5f806a57964a-extensions-socket-volume" pod "e1b2b72a-588c-4847-8963-5f806a57964a" (UID: "e1b2b72a-588c-4847-8963-5f806a57964a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:06.432247 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:06.432223 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1b2b72a-588c-4847-8963-5f806a57964a-kube-api-access-s42wn" pod "e1b2b72a-588c-4847-8963-5f806a57964a" (UID: "e1b2b72a-588c-4847-8963-5f806a57964a"). InnerVolumeSpecName "kube-api-access-s42wn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:06.446156 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:06.446129 4965 generic.go:356] "Generic (PLEG): container finished" podID="e1b2b72a-588c-4847-8963-5f806a57964a" containerID="16329df6da7256b6fa84b2eb74e8e63b25b3adb8ed04aa974a8c016979f3f1e8" exitCode=0 Oct 30 17:21:06.446295 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:06.446196 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-v8fjn" Oct 30 17:21:06.446295 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:06.446199 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-v8fjn" event={"ID":"e1b2b72a-588c-4847-8963-5f806a57964a","Type":"ContainerDied","Data":"16329df6da7256b6fa84b2eb74e8e63b25b3adb8ed04aa974a8c016979f3f1e8"} Oct 30 17:21:06.446295 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:06.446256 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-v8fjn" event={"ID":"e1b2b72a-588c-4847-8963-5f806a57964a","Type":"ContainerDied","Data":"3a8011ca2e51bb0b5ca20929c8ee05eea63cfd3a8d46f7fbada6f6d41ef14b2b"} Oct 30 17:21:06.446295 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:06.446279 4965 scope.go:122] "RemoveContainer" containerID="16329df6da7256b6fa84b2eb74e8e63b25b3adb8ed04aa974a8c016979f3f1e8" Oct 30 17:21:06.456005 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:06.455986 4965 scope.go:122] "RemoveContainer" containerID="16329df6da7256b6fa84b2eb74e8e63b25b3adb8ed04aa974a8c016979f3f1e8" Oct 30 17:21:06.456268 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:21:06.456245 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16329df6da7256b6fa84b2eb74e8e63b25b3adb8ed04aa974a8c016979f3f1e8\": container with ID starting with 16329df6da7256b6fa84b2eb74e8e63b25b3adb8ed04aa974a8c016979f3f1e8 not found: ID does not exist" containerID="16329df6da7256b6fa84b2eb74e8e63b25b3adb8ed04aa974a8c016979f3f1e8" Oct 30 17:21:06.456372 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:06.456275 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16329df6da7256b6fa84b2eb74e8e63b25b3adb8ed04aa974a8c016979f3f1e8"} err="failed to get container status \"16329df6da7256b6fa84b2eb74e8e63b25b3adb8ed04aa974a8c016979f3f1e8\": rpc error: code = NotFound desc = could not find container \"16329df6da7256b6fa84b2eb74e8e63b25b3adb8ed04aa974a8c016979f3f1e8\": container with ID starting with 16329df6da7256b6fa84b2eb74e8e63b25b3adb8ed04aa974a8c016979f3f1e8 not found: ID does not exist" Oct 30 17:21:06.500460 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:06.500387 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-v8fjn"] Oct 30 17:21:06.507361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:06.507338 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-v8fjn"] Oct 30 17:21:06.531310 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:06.531286 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s42wn\" (UniqueName: \"kubernetes.io/projected/e1b2b72a-588c-4847-8963-5f806a57964a-kube-api-access-s42wn\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:21:06.531310 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:06.531308 4965 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1b2b72a-588c-4847-8963-5f806a57964a-extensions-socket-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:21:08.096815 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:08.096776 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1b2b72a-588c-4847-8963-5f806a57964a" path="/var/lib/kubelet/pods/e1b2b72a-588c-4847-8963-5f806a57964a/volumes" Oct 30 17:21:25.882362 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:25.882328 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8zvlz"] Oct 30 17:21:25.882733 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:25.882661 4965 cpu_manager.go:414] "containerMap: removing container" podUID="6ebae7e5-7c84-41df-882c-a130a210496f" containerName="manager" Oct 30 17:21:25.882733 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:25.882671 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ebae7e5-7c84-41df-882c-a130a210496f" containerName="manager" Oct 30 17:21:25.882733 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:25.882682 4965 cpu_manager.go:414] "containerMap: removing container" podUID="e1b2b72a-588c-4847-8963-5f806a57964a" containerName="manager" Oct 30 17:21:25.882733 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:25.882688 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1b2b72a-588c-4847-8963-5f806a57964a" containerName="manager" Oct 30 17:21:25.882913 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:25.882757 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ebae7e5-7c84-41df-882c-a130a210496f" containerName="manager" Oct 30 17:21:25.882913 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:25.882764 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1b2b72a-588c-4847-8963-5f806a57964a" containerName="manager" Oct 30 17:21:25.887178 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:25.887158 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-8zvlz" Oct 30 17:21:25.903774 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:25.903744 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8zvlz"] Oct 30 17:21:25.914346 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:25.914318 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 17:21:25.914461 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:25.914359 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2wkrd\"" Oct 30 17:21:26.004083 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:26.004051 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/855a7736-024f-4389-b26d-6501d751f97f-config-file\") pod \"limitador-limitador-74dfcfb894-8zvlz\" (UID: \"855a7736-024f-4389-b26d-6501d751f97f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-8zvlz" Oct 30 17:21:26.004251 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:26.004108 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv4rh\" (UniqueName: \"kubernetes.io/projected/855a7736-024f-4389-b26d-6501d751f97f-kube-api-access-mv4rh\") pod \"limitador-limitador-74dfcfb894-8zvlz\" (UID: \"855a7736-024f-4389-b26d-6501d751f97f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-8zvlz" Oct 30 17:21:26.015955 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:26.015923 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8zvlz"] Oct 30 17:21:26.105438 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:26.105397 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/855a7736-024f-4389-b26d-6501d751f97f-config-file\") pod \"limitador-limitador-74dfcfb894-8zvlz\" (UID: \"855a7736-024f-4389-b26d-6501d751f97f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-8zvlz" Oct 30 17:21:26.105644 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:26.105476 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv4rh\" (UniqueName: \"kubernetes.io/projected/855a7736-024f-4389-b26d-6501d751f97f-kube-api-access-mv4rh\") pod \"limitador-limitador-74dfcfb894-8zvlz\" (UID: \"855a7736-024f-4389-b26d-6501d751f97f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-8zvlz" Oct 30 17:21:26.106179 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:26.106148 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/855a7736-024f-4389-b26d-6501d751f97f-config-file\") pod \"limitador-limitador-74dfcfb894-8zvlz\" (UID: \"855a7736-024f-4389-b26d-6501d751f97f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-8zvlz" Oct 30 17:21:26.129005 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:26.128969 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv4rh\" (UniqueName: \"kubernetes.io/projected/855a7736-024f-4389-b26d-6501d751f97f-kube-api-access-mv4rh\") pod \"limitador-limitador-74dfcfb894-8zvlz\" (UID: \"855a7736-024f-4389-b26d-6501d751f97f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-8zvlz" Oct 30 17:21:26.200077 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:26.199988 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-8zvlz" Oct 30 17:21:26.502658 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:26.502630 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8zvlz"] Oct 30 17:21:26.504852 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:21:26.504817 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod855a7736_024f_4389_b26d_6501d751f97f.slice/crio-abb3e594f443f231aa6219adb7f8106be17c85ec8845065f656ccf10fcac4863 WatchSource:0}: Error finding container abb3e594f443f231aa6219adb7f8106be17c85ec8845065f656ccf10fcac4863: Status 404 returned error can't find the container with id abb3e594f443f231aa6219adb7f8106be17c85ec8845065f656ccf10fcac4863 Oct 30 17:21:26.534236 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:26.534204 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-8zvlz" event={"ID":"855a7736-024f-4389-b26d-6501d751f97f","Type":"ContainerStarted","Data":"abb3e594f443f231aa6219adb7f8106be17c85ec8845065f656ccf10fcac4863"} Oct 30 17:21:26.745520 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:26.743093 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:21:26.748813 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:26.748790 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-kqf65" Oct 30 17:21:26.763833 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:26.763805 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:21:26.836778 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:26.836746 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:21:26.911518 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:26.911484 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/31a77224-5836-451e-a601-0229e81d4e74-config-file\") pod \"limitador-limitador-7c9b87d574-kqf65\" (UID: \"31a77224-5836-451e-a601-0229e81d4e74\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-kqf65" Oct 30 17:21:26.911909 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:26.911539 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tdjn\" (UniqueName: \"kubernetes.io/projected/31a77224-5836-451e-a601-0229e81d4e74-kube-api-access-2tdjn\") pod \"limitador-limitador-7c9b87d574-kqf65\" (UID: \"31a77224-5836-451e-a601-0229e81d4e74\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-kqf65" Oct 30 17:21:27.012272 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:27.012197 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tdjn\" (UniqueName: \"kubernetes.io/projected/31a77224-5836-451e-a601-0229e81d4e74-kube-api-access-2tdjn\") pod \"limitador-limitador-7c9b87d574-kqf65\" (UID: \"31a77224-5836-451e-a601-0229e81d4e74\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-kqf65" Oct 30 17:21:27.012404 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:27.012282 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/31a77224-5836-451e-a601-0229e81d4e74-config-file\") pod \"limitador-limitador-7c9b87d574-kqf65\" (UID: \"31a77224-5836-451e-a601-0229e81d4e74\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-kqf65" Oct 30 17:21:27.012930 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:27.012908 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/31a77224-5836-451e-a601-0229e81d4e74-config-file\") pod \"limitador-limitador-7c9b87d574-kqf65\" (UID: \"31a77224-5836-451e-a601-0229e81d4e74\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-kqf65" Oct 30 17:21:27.035990 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:27.035959 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tdjn\" (UniqueName: \"kubernetes.io/projected/31a77224-5836-451e-a601-0229e81d4e74-kube-api-access-2tdjn\") pod \"limitador-limitador-7c9b87d574-kqf65\" (UID: \"31a77224-5836-451e-a601-0229e81d4e74\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-kqf65" Oct 30 17:21:27.061704 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:27.061680 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-kqf65" Oct 30 17:21:27.357358 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:27.357335 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:21:27.359448 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:21:27.359423 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31a77224_5836_451e_a601_0229e81d4e74.slice/crio-0f83524147685b98229126e271a0fe7ea78f9527cb3fe67d56ea64403f97cce5 WatchSource:0}: Error finding container 0f83524147685b98229126e271a0fe7ea78f9527cb3fe67d56ea64403f97cce5: Status 404 returned error can't find the container with id 0f83524147685b98229126e271a0fe7ea78f9527cb3fe67d56ea64403f97cce5 Oct 30 17:21:27.539106 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:27.539074 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-kqf65" event={"ID":"31a77224-5836-451e-a601-0229e81d4e74","Type":"ContainerStarted","Data":"0f83524147685b98229126e271a0fe7ea78f9527cb3fe67d56ea64403f97cce5"} Oct 30 17:21:28.546832 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:28.546799 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-kqf65" event={"ID":"31a77224-5836-451e-a601-0229e81d4e74","Type":"ContainerStarted","Data":"b303180270844aeb60e4b5891929a5f55c4f690a1b44fa4aa34ab919237d3f9e"} Oct 30 17:21:28.547252 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:28.546919 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-kqf65" Oct 30 17:21:28.548557 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:28.548534 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-8zvlz" event={"ID":"855a7736-024f-4389-b26d-6501d751f97f","Type":"ContainerStarted","Data":"2b872590d9127b04b1abfea2978bbd396c7eba9ca042822af17e67ec49c70fb0"} Oct 30 17:21:28.548680 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:28.548601 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-8zvlz" Oct 30 17:21:28.589442 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:28.589391 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-kqf65" podStartSLOduration=1.518356337 podStartE2EDuration="2.589379233s" totalImagesPullingTime="1.071022896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:21:28.588186205 +0000 UTC m=+546.640286154" watchObservedRunningTime="2026-10-30 17:21:28.589379233 +0000 UTC m=+546.641479164" Oct 30 17:21:28.629990 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:28.629942 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-8zvlz" podStartSLOduration=2.140502603 podStartE2EDuration="3.629928318s" totalImagesPullingTime="1.489425715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:21:28.628923451 +0000 UTC m=+546.681023401" watchObservedRunningTime="2026-10-30 17:21:28.629928318 +0000 UTC m=+546.682028290" Oct 30 17:21:35.995669 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:21:35.995628 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:21:39.554402 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:39.554374 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-8zvlz" Oct 30 17:21:39.554917 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:39.554441 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-kqf65" Oct 30 17:21:39.628034 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:39.628005 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8zvlz"] Oct 30 17:21:39.628195 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:39.628159 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-8zvlz" podUID="855a7736-024f-4389-b26d-6501d751f97f" containerName="limitador" containerID="cri-o://2b872590d9127b04b1abfea2978bbd396c7eba9ca042822af17e67ec49c70fb0" gracePeriod=30 Oct 30 17:21:40.174279 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:40.174255 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-8zvlz" Oct 30 17:21:40.329149 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:40.329121 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/855a7736-024f-4389-b26d-6501d751f97f-kube-api-access-mv4rh\" (UniqueName: \"kubernetes.io/projected/855a7736-024f-4389-b26d-6501d751f97f-kube-api-access-mv4rh\") pod \"855a7736-024f-4389-b26d-6501d751f97f\" (UID: \"855a7736-024f-4389-b26d-6501d751f97f\") " Oct 30 17:21:40.329324 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:40.329177 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/855a7736-024f-4389-b26d-6501d751f97f-config-file\" (UniqueName: \"kubernetes.io/configmap/855a7736-024f-4389-b26d-6501d751f97f-config-file\") pod \"855a7736-024f-4389-b26d-6501d751f97f\" (UID: \"855a7736-024f-4389-b26d-6501d751f97f\") " Oct 30 17:21:40.329551 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:40.329528 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/855a7736-024f-4389-b26d-6501d751f97f-config-file" pod "855a7736-024f-4389-b26d-6501d751f97f" (UID: "855a7736-024f-4389-b26d-6501d751f97f"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:40.331212 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:40.331189 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/855a7736-024f-4389-b26d-6501d751f97f-kube-api-access-mv4rh" pod "855a7736-024f-4389-b26d-6501d751f97f" (UID: "855a7736-024f-4389-b26d-6501d751f97f"). InnerVolumeSpecName "kube-api-access-mv4rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:40.430442 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:40.430419 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv4rh\" (UniqueName: \"kubernetes.io/projected/855a7736-024f-4389-b26d-6501d751f97f-kube-api-access-mv4rh\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:21:40.430442 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:40.430440 4965 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/855a7736-024f-4389-b26d-6501d751f97f-config-file\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:21:40.600795 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:40.600714 4965 generic.go:356] "Generic (PLEG): container finished" podID="855a7736-024f-4389-b26d-6501d751f97f" containerID="2b872590d9127b04b1abfea2978bbd396c7eba9ca042822af17e67ec49c70fb0" exitCode=0 Oct 30 17:21:40.600795 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:40.600753 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-8zvlz" event={"ID":"855a7736-024f-4389-b26d-6501d751f97f","Type":"ContainerDied","Data":"2b872590d9127b04b1abfea2978bbd396c7eba9ca042822af17e67ec49c70fb0"} Oct 30 17:21:40.600795 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:40.600778 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-8zvlz" Oct 30 17:21:40.601353 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:40.600815 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-8zvlz" event={"ID":"855a7736-024f-4389-b26d-6501d751f97f","Type":"ContainerDied","Data":"abb3e594f443f231aa6219adb7f8106be17c85ec8845065f656ccf10fcac4863"} Oct 30 17:21:40.601353 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:40.600836 4965 scope.go:122] "RemoveContainer" containerID="2b872590d9127b04b1abfea2978bbd396c7eba9ca042822af17e67ec49c70fb0" Oct 30 17:21:40.611144 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:40.611110 4965 scope.go:122] "RemoveContainer" containerID="2b872590d9127b04b1abfea2978bbd396c7eba9ca042822af17e67ec49c70fb0" Oct 30 17:21:40.611508 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:21:40.611478 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b872590d9127b04b1abfea2978bbd396c7eba9ca042822af17e67ec49c70fb0\": container with ID starting with 2b872590d9127b04b1abfea2978bbd396c7eba9ca042822af17e67ec49c70fb0 not found: ID does not exist" containerID="2b872590d9127b04b1abfea2978bbd396c7eba9ca042822af17e67ec49c70fb0" Oct 30 17:21:40.611603 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:40.611528 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b872590d9127b04b1abfea2978bbd396c7eba9ca042822af17e67ec49c70fb0"} err="failed to get container status \"2b872590d9127b04b1abfea2978bbd396c7eba9ca042822af17e67ec49c70fb0\": rpc error: code = NotFound desc = could not find container \"2b872590d9127b04b1abfea2978bbd396c7eba9ca042822af17e67ec49c70fb0\": container with ID starting with 2b872590d9127b04b1abfea2978bbd396c7eba9ca042822af17e67ec49c70fb0 not found: ID does not exist" Oct 30 17:21:40.655802 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:40.655776 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8zvlz"] Oct 30 17:21:40.667878 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:40.667830 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8zvlz"] Oct 30 17:21:42.097928 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:42.097888 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="855a7736-024f-4389-b26d-6501d751f97f" path="/var/lib/kubelet/pods/855a7736-024f-4389-b26d-6501d751f97f/volumes" Oct 30 17:21:45.780162 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:45.780127 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-pd82z"] Oct 30 17:21:45.780657 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:45.780636 4965 cpu_manager.go:414] "containerMap: removing container" podUID="855a7736-024f-4389-b26d-6501d751f97f" containerName="limitador" Oct 30 17:21:45.780730 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:45.780659 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="855a7736-024f-4389-b26d-6501d751f97f" containerName="limitador" Oct 30 17:21:45.780801 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:45.780788 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="855a7736-024f-4389-b26d-6501d751f97f" containerName="limitador" Oct 30 17:21:45.785148 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:45.785126 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pd82z" Oct 30 17:21:45.800891 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:45.800670 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 17:21:45.800891 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:45.800824 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-vfj6z\"" Oct 30 17:21:45.813137 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:45.813084 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:21:45.813137 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:45.813126 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:21:45.814586 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:45.814564 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-pd82z"] Oct 30 17:21:45.872160 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:45.872133 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2ae50ffd-59e5-43e8-8e4a-b8080c9eddc0-data\") pod \"postgres-7cdbf6dc44-pd82z\" (UID: \"2ae50ffd-59e5-43e8-8e4a-b8080c9eddc0\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pd82z" Oct 30 17:21:45.872314 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:45.872170 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wljww\" (UniqueName: \"kubernetes.io/projected/2ae50ffd-59e5-43e8-8e4a-b8080c9eddc0-kube-api-access-wljww\") pod \"postgres-7cdbf6dc44-pd82z\" (UID: \"2ae50ffd-59e5-43e8-8e4a-b8080c9eddc0\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pd82z" Oct 30 17:21:45.973067 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:45.973032 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2ae50ffd-59e5-43e8-8e4a-b8080c9eddc0-data\") pod \"postgres-7cdbf6dc44-pd82z\" (UID: \"2ae50ffd-59e5-43e8-8e4a-b8080c9eddc0\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pd82z" Oct 30 17:21:45.973235 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:45.973077 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wljww\" (UniqueName: \"kubernetes.io/projected/2ae50ffd-59e5-43e8-8e4a-b8080c9eddc0-kube-api-access-wljww\") pod \"postgres-7cdbf6dc44-pd82z\" (UID: \"2ae50ffd-59e5-43e8-8e4a-b8080c9eddc0\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pd82z" Oct 30 17:21:45.973432 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:45.973410 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2ae50ffd-59e5-43e8-8e4a-b8080c9eddc0-data\") pod \"postgres-7cdbf6dc44-pd82z\" (UID: \"2ae50ffd-59e5-43e8-8e4a-b8080c9eddc0\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pd82z" Oct 30 17:21:45.996877 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:45.996836 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wljww\" (UniqueName: \"kubernetes.io/projected/2ae50ffd-59e5-43e8-8e4a-b8080c9eddc0-kube-api-access-wljww\") pod \"postgres-7cdbf6dc44-pd82z\" (UID: \"2ae50ffd-59e5-43e8-8e4a-b8080c9eddc0\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pd82z" Oct 30 17:21:46.097061 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:46.097032 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pd82z" Oct 30 17:21:46.402814 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:46.402791 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-pd82z"] Oct 30 17:21:46.404951 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:21:46.404924 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae50ffd_59e5_43e8_8e4a_b8080c9eddc0.slice/crio-eb16db11d16458efcb20fc98fc2a10d4941e28dce5daaa452126eb1fbf29c664 WatchSource:0}: Error finding container eb16db11d16458efcb20fc98fc2a10d4941e28dce5daaa452126eb1fbf29c664: Status 404 returned error can't find the container with id eb16db11d16458efcb20fc98fc2a10d4941e28dce5daaa452126eb1fbf29c664 Oct 30 17:21:46.627172 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:46.627135 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pd82z" event={"ID":"2ae50ffd-59e5-43e8-8e4a-b8080c9eddc0","Type":"ContainerStarted","Data":"eb16db11d16458efcb20fc98fc2a10d4941e28dce5daaa452126eb1fbf29c664"} Oct 30 17:21:51.654615 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:51.654577 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pd82z" event={"ID":"2ae50ffd-59e5-43e8-8e4a-b8080c9eddc0","Type":"ContainerStarted","Data":"83a417df9c67e23192580feaac09d5a6e739bac2f46e1f60e1b974c97c5f9edb"} Oct 30 17:21:51.655032 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:51.654712 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pd82z" Oct 30 17:21:51.696883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:51.696821 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pd82z" podStartSLOduration=1.926913535 podStartE2EDuration="6.696808946s" totalImagesPullingTime="4.769895411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:21:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:21:51.695388797 +0000 UTC m=+569.747488729" watchObservedRunningTime="2026-10-30 17:21:51.696808946 +0000 UTC m=+569.748908874" Oct 30 17:21:57.687295 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:21:57.687262 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pd82z" Oct 30 17:22:05.996002 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:22:05.995902 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:22:22.145717 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:22:22.145685 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 17:22:22.147431 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:22:22.147404 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 17:22:35.996711 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:22:35.996671 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:23:05.437766 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:23:05.437729 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-6tb6z"] Oct 30 17:23:05.441208 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:23:05.441191 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-6tb6z" Oct 30 17:23:05.472382 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:23:05.472353 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s8xph\"" Oct 30 17:23:05.472583 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:23:05.472389 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 17:23:05.472583 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:23:05.472411 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 17:23:05.474015 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:23:05.473990 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-6tb6z"] Oct 30 17:23:05.552568 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:23:05.552535 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm85w\" (UniqueName: \"kubernetes.io/projected/c2f9d33f-085d-4744-85cc-d309f9f0ccad-kube-api-access-wm85w\") pod \"maas-api-5c48877c75-6tb6z\" (UID: \"c2f9d33f-085d-4744-85cc-d309f9f0ccad\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-6tb6z" Oct 30 17:23:05.552757 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:23:05.552578 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2f9d33f-085d-4744-85cc-d309f9f0ccad-metrics-certs\") pod \"maas-api-5c48877c75-6tb6z\" (UID: \"c2f9d33f-085d-4744-85cc-d309f9f0ccad\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-6tb6z" Oct 30 17:23:05.552757 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:23:05.552597 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2f9d33f-085d-4744-85cc-d309f9f0ccad-maas-api-tls\") pod \"maas-api-5c48877c75-6tb6z\" (UID: \"c2f9d33f-085d-4744-85cc-d309f9f0ccad\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-6tb6z" Oct 30 17:23:05.552757 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:23:05.552672 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2f9d33f-085d-4744-85cc-d309f9f0ccad-openshift-service-ca\") pod \"maas-api-5c48877c75-6tb6z\" (UID: \"c2f9d33f-085d-4744-85cc-d309f9f0ccad\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-6tb6z" Oct 30 17:23:05.653826 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:23:05.653790 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm85w\" (UniqueName: \"kubernetes.io/projected/c2f9d33f-085d-4744-85cc-d309f9f0ccad-kube-api-access-wm85w\") pod \"maas-api-5c48877c75-6tb6z\" (UID: \"c2f9d33f-085d-4744-85cc-d309f9f0ccad\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-6tb6z" Oct 30 17:23:05.654046 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:23:05.653837 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2f9d33f-085d-4744-85cc-d309f9f0ccad-metrics-certs\") pod \"maas-api-5c48877c75-6tb6z\" (UID: \"c2f9d33f-085d-4744-85cc-d309f9f0ccad\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-6tb6z" Oct 30 17:23:05.654046 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:23:05.653855 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2f9d33f-085d-4744-85cc-d309f9f0ccad-maas-api-tls\") pod \"maas-api-5c48877c75-6tb6z\" (UID: \"c2f9d33f-085d-4744-85cc-d309f9f0ccad\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-6tb6z" Oct 30 17:23:05.654046 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:23:05.653982 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2f9d33f-085d-4744-85cc-d309f9f0ccad-openshift-service-ca\") pod \"maas-api-5c48877c75-6tb6z\" (UID: \"c2f9d33f-085d-4744-85cc-d309f9f0ccad\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-6tb6z" Oct 30 17:23:05.654605 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:23:05.654585 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2f9d33f-085d-4744-85cc-d309f9f0ccad-openshift-service-ca\") pod \"maas-api-5c48877c75-6tb6z\" (UID: \"c2f9d33f-085d-4744-85cc-d309f9f0ccad\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-6tb6z" Oct 30 17:23:05.656358 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:23:05.656336 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2f9d33f-085d-4744-85cc-d309f9f0ccad-metrics-certs\") pod \"maas-api-5c48877c75-6tb6z\" (UID: \"c2f9d33f-085d-4744-85cc-d309f9f0ccad\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-6tb6z" Oct 30 17:23:05.656451 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:23:05.656394 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2f9d33f-085d-4744-85cc-d309f9f0ccad-maas-api-tls\") pod \"maas-api-5c48877c75-6tb6z\" (UID: \"c2f9d33f-085d-4744-85cc-d309f9f0ccad\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-6tb6z" Oct 30 17:23:05.677677 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:23:05.677644 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm85w\" (UniqueName: \"kubernetes.io/projected/c2f9d33f-085d-4744-85cc-d309f9f0ccad-kube-api-access-wm85w\") pod \"maas-api-5c48877c75-6tb6z\" (UID: \"c2f9d33f-085d-4744-85cc-d309f9f0ccad\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-6tb6z" Oct 30 17:23:05.754275 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:23:05.754160 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-6tb6z" Oct 30 17:23:05.995693 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:23:05.995657 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:23:06.054694 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:23:06.054643 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-6tb6z"] Oct 30 17:23:06.057013 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:23:06.056986 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f9d33f_085d_4744_85cc_d309f9f0ccad.slice/crio-93638b38513dd17d6016bb49f46472ba45c827c8b2cedd2c1d9a5d16be374743 WatchSource:0}: Error finding container 93638b38513dd17d6016bb49f46472ba45c827c8b2cedd2c1d9a5d16be374743: Status 404 returned error can't find the container with id 93638b38513dd17d6016bb49f46472ba45c827c8b2cedd2c1d9a5d16be374743 Oct 30 17:23:06.058199 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:23:06.058181 4965 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:23:06.933508 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:23:06.932758 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:23:06.983076 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:23:06.983029 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-6tb6z" event={"ID":"c2f9d33f-085d-4744-85cc-d309f9f0ccad","Type":"ContainerStarted","Data":"93638b38513dd17d6016bb49f46472ba45c827c8b2cedd2c1d9a5d16be374743"} Oct 30 17:23:08.994331 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:23:08.994298 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-6tb6z" event={"ID":"c2f9d33f-085d-4744-85cc-d309f9f0ccad","Type":"ContainerStarted","Data":"12f071f9e0f895805ac2ee7dfc3225b7d06c08231a38be734410754a6c2eb9c5"} Oct 30 17:23:08.994793 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:23:08.994437 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-6tb6z" Oct 30 17:23:09.038050 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:23:09.038006 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-6tb6z" podStartSLOduration=1.719917143 podStartE2EDuration="4.03799328s" totalImagesPullingTime="2.318076137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:23:09.036660326 +0000 UTC m=+647.088760269" watchObservedRunningTime="2026-10-30 17:23:09.03799328 +0000 UTC m=+647.090093211" Oct 30 17:23:15.004059 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:23:15.004031 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-6tb6z" Oct 30 17:23:35.995006 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:23:35.994925 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:24:05.996847 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:24:05.996803 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:24:09.188328 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:09.188296 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:24:11.740082 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:11.740051 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-q6l6h"] Oct 30 17:24:11.743613 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:11.743593 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q6l6h" Oct 30 17:24:11.773355 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:11.773317 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hbtff\"" Oct 30 17:24:11.773355 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:11.773317 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 17:24:11.773355 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:11.773347 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 17:24:11.773565 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:11.773368 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 17:24:11.775148 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:11.775027 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-q6l6h"] Oct 30 17:24:11.810490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:11.810463 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6416895-72e8-4440-9d81-f5854fed4aa5-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q6l6h\" (UID: \"c6416895-72e8-4440-9d81-f5854fed4aa5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q6l6h" Oct 30 17:24:11.810606 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:11.810535 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6416895-72e8-4440-9d81-f5854fed4aa5-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q6l6h\" (UID: \"c6416895-72e8-4440-9d81-f5854fed4aa5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q6l6h" Oct 30 17:24:11.810606 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:11.810564 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxmt7\" (UniqueName: \"kubernetes.io/projected/c6416895-72e8-4440-9d81-f5854fed4aa5-kube-api-access-jxmt7\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q6l6h\" (UID: \"c6416895-72e8-4440-9d81-f5854fed4aa5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q6l6h" Oct 30 17:24:11.810606 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:11.810584 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6416895-72e8-4440-9d81-f5854fed4aa5-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q6l6h\" (UID: \"c6416895-72e8-4440-9d81-f5854fed4aa5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q6l6h" Oct 30 17:24:11.810725 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:11.810642 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6416895-72e8-4440-9d81-f5854fed4aa5-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q6l6h\" (UID: \"c6416895-72e8-4440-9d81-f5854fed4aa5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q6l6h" Oct 30 17:24:11.810725 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:11.810697 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6416895-72e8-4440-9d81-f5854fed4aa5-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q6l6h\" (UID: \"c6416895-72e8-4440-9d81-f5854fed4aa5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q6l6h" Oct 30 17:24:11.911843 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:11.911812 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6416895-72e8-4440-9d81-f5854fed4aa5-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q6l6h\" (UID: \"c6416895-72e8-4440-9d81-f5854fed4aa5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q6l6h" Oct 30 17:24:11.911843 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:11.911846 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxmt7\" (UniqueName: \"kubernetes.io/projected/c6416895-72e8-4440-9d81-f5854fed4aa5-kube-api-access-jxmt7\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q6l6h\" (UID: \"c6416895-72e8-4440-9d81-f5854fed4aa5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q6l6h" Oct 30 17:24:11.912072 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:11.911890 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6416895-72e8-4440-9d81-f5854fed4aa5-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q6l6h\" (UID: \"c6416895-72e8-4440-9d81-f5854fed4aa5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q6l6h" Oct 30 17:24:11.912072 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:11.911928 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6416895-72e8-4440-9d81-f5854fed4aa5-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q6l6h\" (UID: \"c6416895-72e8-4440-9d81-f5854fed4aa5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q6l6h" Oct 30 17:24:11.912072 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:11.912052 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6416895-72e8-4440-9d81-f5854fed4aa5-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q6l6h\" (UID: \"c6416895-72e8-4440-9d81-f5854fed4aa5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q6l6h" Oct 30 17:24:11.912217 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:11.912099 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6416895-72e8-4440-9d81-f5854fed4aa5-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q6l6h\" (UID: \"c6416895-72e8-4440-9d81-f5854fed4aa5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q6l6h" Oct 30 17:24:11.912365 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:11.912342 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6416895-72e8-4440-9d81-f5854fed4aa5-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q6l6h\" (UID: \"c6416895-72e8-4440-9d81-f5854fed4aa5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q6l6h" Oct 30 17:24:11.912437 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:11.912351 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6416895-72e8-4440-9d81-f5854fed4aa5-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q6l6h\" (UID: \"c6416895-72e8-4440-9d81-f5854fed4aa5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q6l6h" Oct 30 17:24:11.912437 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:11.912398 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6416895-72e8-4440-9d81-f5854fed4aa5-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q6l6h\" (UID: \"c6416895-72e8-4440-9d81-f5854fed4aa5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q6l6h" Oct 30 17:24:11.914294 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:11.914266 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6416895-72e8-4440-9d81-f5854fed4aa5-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q6l6h\" (UID: \"c6416895-72e8-4440-9d81-f5854fed4aa5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q6l6h" Oct 30 17:24:11.914469 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:11.914450 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6416895-72e8-4440-9d81-f5854fed4aa5-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q6l6h\" (UID: \"c6416895-72e8-4440-9d81-f5854fed4aa5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q6l6h" Oct 30 17:24:11.934956 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:11.934933 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxmt7\" (UniqueName: \"kubernetes.io/projected/c6416895-72e8-4440-9d81-f5854fed4aa5-kube-api-access-jxmt7\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q6l6h\" (UID: \"c6416895-72e8-4440-9d81-f5854fed4aa5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q6l6h" Oct 30 17:24:12.062392 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:12.062312 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q6l6h" Oct 30 17:24:12.360778 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:12.360754 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-q6l6h"] Oct 30 17:24:12.362960 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:24:12.362930 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6416895_72e8_4440_9d81_f5854fed4aa5.slice/crio-f3dde37e4178421cf85266a9d439f5e04fd409bea4a024bfc541056e583654bc WatchSource:0}: Error finding container f3dde37e4178421cf85266a9d439f5e04fd409bea4a024bfc541056e583654bc: Status 404 returned error can't find the container with id f3dde37e4178421cf85266a9d439f5e04fd409bea4a024bfc541056e583654bc Oct 30 17:24:13.273120 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:13.273073 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q6l6h" event={"ID":"c6416895-72e8-4440-9d81-f5854fed4aa5","Type":"ContainerStarted","Data":"f3dde37e4178421cf85266a9d439f5e04fd409bea4a024bfc541056e583654bc"} Oct 30 17:24:13.616931 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:13.616899 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:24:14.278454 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:14.278420 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q6l6h" event={"ID":"c6416895-72e8-4440-9d81-f5854fed4aa5","Type":"ContainerStarted","Data":"b6bc7d92fc7935875acfcc847be90927e8e31ad76ee9c067393e06450097a118"} Oct 30 17:24:14.322364 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:14.322315 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q6l6h" podStartSLOduration=2.276966485 podStartE2EDuration="3.322301846s" totalImagesPullingTime="1.045335361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:14.320881423 +0000 UTC m=+712.372981344" watchObservedRunningTime="2026-10-30 17:24:14.322301846 +0000 UTC m=+712.374401773" Oct 30 17:24:14.511880 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:14.511783 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-st6hg"] Oct 30 17:24:14.515340 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:14.515323 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-st6hg" Oct 30 17:24:14.536727 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:14.536655 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 17:24:14.538221 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:14.538196 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-st6hg"] Oct 30 17:24:14.634677 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:14.634641 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/393bb669-2192-4775-b63c-f96a2aba0cd4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-st6hg\" (UID: \"393bb669-2192-4775-b63c-f96a2aba0cd4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-st6hg" Oct 30 17:24:14.634677 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:14.634677 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/393bb669-2192-4775-b63c-f96a2aba0cd4-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-st6hg\" (UID: \"393bb669-2192-4775-b63c-f96a2aba0cd4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-st6hg" Oct 30 17:24:14.634905 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:14.634700 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/393bb669-2192-4775-b63c-f96a2aba0cd4-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-st6hg\" (UID: \"393bb669-2192-4775-b63c-f96a2aba0cd4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-st6hg" Oct 30 17:24:14.634905 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:14.634823 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/393bb669-2192-4775-b63c-f96a2aba0cd4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-st6hg\" (UID: \"393bb669-2192-4775-b63c-f96a2aba0cd4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-st6hg" Oct 30 17:24:14.634905 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:14.634847 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/393bb669-2192-4775-b63c-f96a2aba0cd4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-st6hg\" (UID: \"393bb669-2192-4775-b63c-f96a2aba0cd4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-st6hg" Oct 30 17:24:14.635010 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:14.634932 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdmcv\" (UniqueName: \"kubernetes.io/projected/393bb669-2192-4775-b63c-f96a2aba0cd4-kube-api-access-kdmcv\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-st6hg\" (UID: \"393bb669-2192-4775-b63c-f96a2aba0cd4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-st6hg" Oct 30 17:24:14.735784 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:14.735749 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/393bb669-2192-4775-b63c-f96a2aba0cd4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-st6hg\" (UID: \"393bb669-2192-4775-b63c-f96a2aba0cd4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-st6hg" Oct 30 17:24:14.735985 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:14.735788 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/393bb669-2192-4775-b63c-f96a2aba0cd4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-st6hg\" (UID: \"393bb669-2192-4775-b63c-f96a2aba0cd4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-st6hg" Oct 30 17:24:14.735985 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:14.735830 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdmcv\" (UniqueName: \"kubernetes.io/projected/393bb669-2192-4775-b63c-f96a2aba0cd4-kube-api-access-kdmcv\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-st6hg\" (UID: \"393bb669-2192-4775-b63c-f96a2aba0cd4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-st6hg" Oct 30 17:24:14.735985 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:14.735901 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/393bb669-2192-4775-b63c-f96a2aba0cd4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-st6hg\" (UID: \"393bb669-2192-4775-b63c-f96a2aba0cd4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-st6hg" Oct 30 17:24:14.735985 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:14.735929 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/393bb669-2192-4775-b63c-f96a2aba0cd4-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-st6hg\" (UID: \"393bb669-2192-4775-b63c-f96a2aba0cd4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-st6hg" Oct 30 17:24:14.735985 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:14.735959 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/393bb669-2192-4775-b63c-f96a2aba0cd4-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-st6hg\" (UID: \"393bb669-2192-4775-b63c-f96a2aba0cd4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-st6hg" Oct 30 17:24:14.736333 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:14.736301 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/393bb669-2192-4775-b63c-f96a2aba0cd4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-st6hg\" (UID: \"393bb669-2192-4775-b63c-f96a2aba0cd4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-st6hg" Oct 30 17:24:14.736438 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:14.736349 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/393bb669-2192-4775-b63c-f96a2aba0cd4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-st6hg\" (UID: \"393bb669-2192-4775-b63c-f96a2aba0cd4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-st6hg" Oct 30 17:24:14.736438 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:14.736390 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/393bb669-2192-4775-b63c-f96a2aba0cd4-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-st6hg\" (UID: \"393bb669-2192-4775-b63c-f96a2aba0cd4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-st6hg" Oct 30 17:24:14.738213 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:14.738191 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/393bb669-2192-4775-b63c-f96a2aba0cd4-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-st6hg\" (UID: \"393bb669-2192-4775-b63c-f96a2aba0cd4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-st6hg" Oct 30 17:24:14.738400 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:14.738382 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/393bb669-2192-4775-b63c-f96a2aba0cd4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-st6hg\" (UID: \"393bb669-2192-4775-b63c-f96a2aba0cd4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-st6hg" Oct 30 17:24:14.767889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:14.767814 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdmcv\" (UniqueName: \"kubernetes.io/projected/393bb669-2192-4775-b63c-f96a2aba0cd4-kube-api-access-kdmcv\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-st6hg\" (UID: \"393bb669-2192-4775-b63c-f96a2aba0cd4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-st6hg" Oct 30 17:24:14.828778 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:14.828748 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-st6hg" Oct 30 17:24:15.062783 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:15.062708 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q6l6h" Oct 30 17:24:15.126888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:15.126849 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-st6hg"] Oct 30 17:24:15.128769 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:24:15.128743 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393bb669_2192_4775_b63c_f96a2aba0cd4.slice/crio-38d726b1361ad4f71f00c00c1271e9476cadf3614cdd699619c2f8a496ca7658 WatchSource:0}: Error finding container 38d726b1361ad4f71f00c00c1271e9476cadf3614cdd699619c2f8a496ca7658: Status 404 returned error can't find the container with id 38d726b1361ad4f71f00c00c1271e9476cadf3614cdd699619c2f8a496ca7658 Oct 30 17:24:15.283572 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:15.283533 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-st6hg" event={"ID":"393bb669-2192-4775-b63c-f96a2aba0cd4","Type":"ContainerStarted","Data":"38d726b1361ad4f71f00c00c1271e9476cadf3614cdd699619c2f8a496ca7658"} Oct 30 17:24:16.289559 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:16.289511 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-st6hg" event={"ID":"393bb669-2192-4775-b63c-f96a2aba0cd4","Type":"ContainerStarted","Data":"71e2c9a07ac8589f97140a76ea908080123daa4387b557d226f89b86de571a90"} Oct 30 17:24:16.334519 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:16.334467 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-st6hg" podStartSLOduration=2.116779635 podStartE2EDuration="2.334451525s" totalImagesPullingTime="217.67189ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:16.333347188 +0000 UTC m=+714.385447122" watchObservedRunningTime="2026-10-30 17:24:16.334451525 +0000 UTC m=+714.386551457" Oct 30 17:24:16.829114 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:16.829078 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-st6hg" Oct 30 17:24:18.433529 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:18.433494 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:24:18.973088 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:18.973056 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs"] Oct 30 17:24:18.976695 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:18.976676 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs" Oct 30 17:24:18.999551 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:18.999519 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs"] Oct 30 17:24:19.006071 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:19.006049 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 17:24:19.076540 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:19.076509 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8d199b0-9f65-4d32-aafb-62e8dc216076-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs\" (UID: \"d8d199b0-9f65-4d32-aafb-62e8dc216076\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs" Oct 30 17:24:19.076747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:19.076547 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4n72\" (UniqueName: \"kubernetes.io/projected/d8d199b0-9f65-4d32-aafb-62e8dc216076-kube-api-access-f4n72\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs\" (UID: \"d8d199b0-9f65-4d32-aafb-62e8dc216076\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs" Oct 30 17:24:19.076747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:19.076572 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8d199b0-9f65-4d32-aafb-62e8dc216076-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs\" (UID: \"d8d199b0-9f65-4d32-aafb-62e8dc216076\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs" Oct 30 17:24:19.076747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:19.076712 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8d199b0-9f65-4d32-aafb-62e8dc216076-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs\" (UID: \"d8d199b0-9f65-4d32-aafb-62e8dc216076\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs" Oct 30 17:24:19.076910 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:19.076751 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8d199b0-9f65-4d32-aafb-62e8dc216076-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs\" (UID: \"d8d199b0-9f65-4d32-aafb-62e8dc216076\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs" Oct 30 17:24:19.076910 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:19.076810 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8d199b0-9f65-4d32-aafb-62e8dc216076-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs\" (UID: \"d8d199b0-9f65-4d32-aafb-62e8dc216076\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs" Oct 30 17:24:19.177947 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:19.177917 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8d199b0-9f65-4d32-aafb-62e8dc216076-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs\" (UID: \"d8d199b0-9f65-4d32-aafb-62e8dc216076\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs" Oct 30 17:24:19.178139 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:19.177960 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4n72\" (UniqueName: \"kubernetes.io/projected/d8d199b0-9f65-4d32-aafb-62e8dc216076-kube-api-access-f4n72\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs\" (UID: \"d8d199b0-9f65-4d32-aafb-62e8dc216076\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs" Oct 30 17:24:19.178139 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:19.177994 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8d199b0-9f65-4d32-aafb-62e8dc216076-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs\" (UID: \"d8d199b0-9f65-4d32-aafb-62e8dc216076\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs" Oct 30 17:24:19.178139 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:19.178047 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8d199b0-9f65-4d32-aafb-62e8dc216076-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs\" (UID: \"d8d199b0-9f65-4d32-aafb-62e8dc216076\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs" Oct 30 17:24:19.178139 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:19.178072 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8d199b0-9f65-4d32-aafb-62e8dc216076-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs\" (UID: \"d8d199b0-9f65-4d32-aafb-62e8dc216076\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs" Oct 30 17:24:19.178139 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:19.178097 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8d199b0-9f65-4d32-aafb-62e8dc216076-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs\" (UID: \"d8d199b0-9f65-4d32-aafb-62e8dc216076\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs" Oct 30 17:24:19.178462 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:19.178436 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8d199b0-9f65-4d32-aafb-62e8dc216076-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs\" (UID: \"d8d199b0-9f65-4d32-aafb-62e8dc216076\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs" Oct 30 17:24:19.178522 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:19.178489 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8d199b0-9f65-4d32-aafb-62e8dc216076-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs\" (UID: \"d8d199b0-9f65-4d32-aafb-62e8dc216076\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs" Oct 30 17:24:19.178623 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:19.178600 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8d199b0-9f65-4d32-aafb-62e8dc216076-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs\" (UID: \"d8d199b0-9f65-4d32-aafb-62e8dc216076\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs" Oct 30 17:24:19.180311 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:19.180283 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8d199b0-9f65-4d32-aafb-62e8dc216076-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs\" (UID: \"d8d199b0-9f65-4d32-aafb-62e8dc216076\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs" Oct 30 17:24:19.180563 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:19.180544 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8d199b0-9f65-4d32-aafb-62e8dc216076-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs\" (UID: \"d8d199b0-9f65-4d32-aafb-62e8dc216076\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs" Oct 30 17:24:19.202912 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:19.202886 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4n72\" (UniqueName: \"kubernetes.io/projected/d8d199b0-9f65-4d32-aafb-62e8dc216076-kube-api-access-f4n72\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs\" (UID: \"d8d199b0-9f65-4d32-aafb-62e8dc216076\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs" Oct 30 17:24:19.291701 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:19.291622 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs" Oct 30 17:24:19.591401 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:19.591373 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs"] Oct 30 17:24:19.593035 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:24:19.593007 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8d199b0_9f65_4d32_aafb_62e8dc216076.slice/crio-a3f05f069ceacd47b040ae28e57a6a27f85c3d29ebdd747f5ad1037942a1c3f3 WatchSource:0}: Error finding container a3f05f069ceacd47b040ae28e57a6a27f85c3d29ebdd747f5ad1037942a1c3f3: Status 404 returned error can't find the container with id a3f05f069ceacd47b040ae28e57a6a27f85c3d29ebdd747f5ad1037942a1c3f3 Oct 30 17:24:20.308953 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:20.308844 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs" event={"ID":"d8d199b0-9f65-4d32-aafb-62e8dc216076","Type":"ContainerStarted","Data":"29c56d242d8639be13802675aa9cac58f185aac1649ba846aca22fbed02e7170"} Oct 30 17:24:20.308953 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:20.308910 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs" event={"ID":"d8d199b0-9f65-4d32-aafb-62e8dc216076","Type":"ContainerStarted","Data":"a3f05f069ceacd47b040ae28e57a6a27f85c3d29ebdd747f5ad1037942a1c3f3"} Oct 30 17:24:20.351909 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:20.351831 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs" podStartSLOduration=2.131327898 podStartE2EDuration="2.351818622s" totalImagesPullingTime="220.490724ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:20.350393991 +0000 UTC m=+718.402493941" watchObservedRunningTime="2026-10-30 17:24:20.351818622 +0000 UTC m=+718.403918553" Oct 30 17:24:21.292336 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:21.292300 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs" Oct 30 17:24:22.063005 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:22.062967 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q6l6h" Oct 30 17:24:22.076768 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:22.076743 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q6l6h" Oct 30 17:24:22.335118 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:22.335092 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q6l6h" Oct 30 17:24:24.829311 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:24.829279 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-st6hg" Oct 30 17:24:24.841858 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:24.841830 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-st6hg" Oct 30 17:24:25.132224 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:25.132127 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:24:25.354024 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:25.353997 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-st6hg" Oct 30 17:24:26.890986 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:26.890951 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5"] Oct 30 17:24:26.894773 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:26.894753 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:26.920312 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:26.920283 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5"] Oct 30 17:24:26.971688 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:26.971658 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-mccnk\"" Oct 30 17:24:27.048027 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:27.047994 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-ms6c5\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:27.048027 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:27.048031 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-ms6c5\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:27.048256 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:27.048050 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-ms6c5\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:27.048256 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:27.048066 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkghc\" (UniqueName: \"kubernetes.io/projected/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-kube-api-access-wkghc\") pod \"maas-default-gateway-openshift-default-7df854bf67-ms6c5\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:27.048256 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:27.048174 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-ms6c5\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:27.048256 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:27.048208 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-ms6c5\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:27.048256 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:27.048231 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-ms6c5\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:27.048521 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:27.048315 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-ms6c5\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:27.048521 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:27.048359 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-ms6c5\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:27.149339 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:27.149256 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-ms6c5\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:27.149339 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:27.149298 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-ms6c5\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:27.149339 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:27.149318 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-ms6c5\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:27.149591 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:27.149363 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-ms6c5\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:27.149591 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:27.149415 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-ms6c5\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:27.149591 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:27.149479 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-ms6c5\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:27.149591 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:27.149501 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-ms6c5\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:27.149591 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:27.149528 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-ms6c5\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:27.149591 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:27.149552 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkghc\" (UniqueName: \"kubernetes.io/projected/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-kube-api-access-wkghc\") pod \"maas-default-gateway-openshift-default-7df854bf67-ms6c5\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:27.149922 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:27.149703 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-ms6c5\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:27.149922 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:27.149737 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-ms6c5\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:27.150044 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:27.150000 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-ms6c5\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:27.150095 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:27.150044 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-ms6c5\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:27.150280 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:27.150261 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-ms6c5\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:27.151634 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:27.151614 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-ms6c5\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:27.151951 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:27.151931 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-ms6c5\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:27.172685 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:27.172658 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkghc\" (UniqueName: \"kubernetes.io/projected/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-kube-api-access-wkghc\") pod \"maas-default-gateway-openshift-default-7df854bf67-ms6c5\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:27.172788 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:27.172730 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-ms6c5\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:27.207385 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:27.207360 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:27.533913 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:27.533890 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5"] Oct 30 17:24:27.535783 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:24:27.535758 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf835cdef_6be0_4f7d_b4c1_6e5a771e8616.slice/crio-70508f315a352918865d25f49f2a68f2b4b564725ca426efd15a0bb51c33cb5b WatchSource:0}: Error finding container 70508f315a352918865d25f49f2a68f2b4b564725ca426efd15a0bb51c33cb5b: Status 404 returned error can't find the container with id 70508f315a352918865d25f49f2a68f2b4b564725ca426efd15a0bb51c33cb5b Oct 30 17:24:28.352506 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:28.352471 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" event={"ID":"f835cdef-6be0-4f7d-b4c1-6e5a771e8616","Type":"ContainerStarted","Data":"70508f315a352918865d25f49f2a68f2b4b564725ca426efd15a0bb51c33cb5b"} Oct 30 17:24:29.292880 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:29.292827 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs" Oct 30 17:24:29.306920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:29.306891 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs" Oct 30 17:24:29.370254 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:29.370220 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs" Oct 30 17:24:30.088336 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:30.088302 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:24:30.088435 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:30.088372 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:24:30.088435 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:30.088402 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:24:30.364908 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:30.364806 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" event={"ID":"f835cdef-6be0-4f7d-b4c1-6e5a771e8616","Type":"ContainerStarted","Data":"a850a12f8d82f7f3b341312d2de62c9ae3b7cc87fc99a557b76837c8ad24cc1b"} Oct 30 17:24:30.410498 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:30.410420 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" podStartSLOduration=1.860774931 podStartE2EDuration="4.410405993s" totalImagesPullingTime="2.549631062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:30.409809511 +0000 UTC m=+728.461909442" watchObservedRunningTime="2026-10-30 17:24:30.410405993 +0000 UTC m=+728.462505923" Oct 30 17:24:31.207967 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:31.207935 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:31.209463 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:31.209437 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ms6c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 17:24:31.209568 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:31.209482 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" podUID="f835cdef-6be0-4f7d-b4c1-6e5a771e8616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 17:24:32.208729 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:32.208689 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ms6c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 17:24:32.209153 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:32.208776 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" podUID="f835cdef-6be0-4f7d-b4c1-6e5a771e8616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 17:24:33.208086 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:33.208047 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ms6c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 17:24:33.208285 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:33.208102 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" podUID="f835cdef-6be0-4f7d-b4c1-6e5a771e8616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 17:24:34.208221 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:34.208181 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ms6c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 17:24:34.208595 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:34.208235 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" podUID="f835cdef-6be0-4f7d-b4c1-6e5a771e8616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 17:24:34.318563 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:34.318533 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5477499c6-b5bnw"] Oct 30 17:24:34.324772 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:34.324750 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5477499c6-b5bnw" Oct 30 17:24:34.343036 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:34.343009 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5477499c6-b5bnw"] Oct 30 17:24:34.353061 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:34.353037 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-x52p7\"" Oct 30 17:24:34.353170 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:34.353100 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 17:24:34.420424 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:34.420390 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvkcv\" (UniqueName: \"kubernetes.io/projected/18a610f0-be91-40cb-900e-7e009f041fed-kube-api-access-tvkcv\") pod \"ai-gateway-controller-5477499c6-b5bnw\" (UID: \"18a610f0-be91-40cb-900e-7e009f041fed\") " pod="opendatahub/ai-gateway-controller-5477499c6-b5bnw" Oct 30 17:24:34.420579 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:34.420437 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18a610f0-be91-40cb-900e-7e009f041fed-metrics-certs\") pod \"ai-gateway-controller-5477499c6-b5bnw\" (UID: \"18a610f0-be91-40cb-900e-7e009f041fed\") " pod="opendatahub/ai-gateway-controller-5477499c6-b5bnw" Oct 30 17:24:34.441412 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:34.441382 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 17:24:34.521730 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:34.521647 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvkcv\" (UniqueName: \"kubernetes.io/projected/18a610f0-be91-40cb-900e-7e009f041fed-kube-api-access-tvkcv\") pod \"ai-gateway-controller-5477499c6-b5bnw\" (UID: \"18a610f0-be91-40cb-900e-7e009f041fed\") " pod="opendatahub/ai-gateway-controller-5477499c6-b5bnw" Oct 30 17:24:34.521730 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:34.521711 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18a610f0-be91-40cb-900e-7e009f041fed-metrics-certs\") pod \"ai-gateway-controller-5477499c6-b5bnw\" (UID: \"18a610f0-be91-40cb-900e-7e009f041fed\") " pod="opendatahub/ai-gateway-controller-5477499c6-b5bnw" Oct 30 17:24:34.524181 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:34.524147 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18a610f0-be91-40cb-900e-7e009f041fed-metrics-certs\") pod \"ai-gateway-controller-5477499c6-b5bnw\" (UID: \"18a610f0-be91-40cb-900e-7e009f041fed\") " pod="opendatahub/ai-gateway-controller-5477499c6-b5bnw" Oct 30 17:24:34.545595 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:34.545562 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvkcv\" (UniqueName: \"kubernetes.io/projected/18a610f0-be91-40cb-900e-7e009f041fed-kube-api-access-tvkcv\") pod \"ai-gateway-controller-5477499c6-b5bnw\" (UID: \"18a610f0-be91-40cb-900e-7e009f041fed\") " pod="opendatahub/ai-gateway-controller-5477499c6-b5bnw" Oct 30 17:24:34.641604 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:34.641567 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5477499c6-b5bnw" Oct 30 17:24:34.976888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:34.967777 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5477499c6-b5bnw"] Oct 30 17:24:34.985259 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:24:34.985231 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a610f0_be91_40cb_900e_7e009f041fed.slice/crio-92d074abe023106b7c7d7fb463bfdae75b1f25545c35a601a1ca87b7d4ca4c22 WatchSource:0}: Error finding container 92d074abe023106b7c7d7fb463bfdae75b1f25545c35a601a1ca87b7d4ca4c22: Status 404 returned error can't find the container with id 92d074abe023106b7c7d7fb463bfdae75b1f25545c35a601a1ca87b7d4ca4c22 Oct 30 17:24:35.207764 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:35.207731 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ms6c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 17:24:35.207964 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:35.207781 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" podUID="f835cdef-6be0-4f7d-b4c1-6e5a771e8616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 17:24:35.404514 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:35.404475 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5477499c6-b5bnw" event={"ID":"18a610f0-be91-40cb-900e-7e009f041fed","Type":"ContainerStarted","Data":"92d074abe023106b7c7d7fb463bfdae75b1f25545c35a601a1ca87b7d4ca4c22"} Oct 30 17:24:36.017400 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:24:36.017355 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:24:36.208561 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:36.208135 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ms6c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 17:24:36.208561 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:36.208196 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" podUID="f835cdef-6be0-4f7d-b4c1-6e5a771e8616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 17:24:37.207816 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:37.207740 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ms6c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 17:24:37.207816 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:37.207803 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" podUID="f835cdef-6be0-4f7d-b4c1-6e5a771e8616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 17:24:38.421216 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:38.421177 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5477499c6-b5bnw" event={"ID":"18a610f0-be91-40cb-900e-7e009f041fed","Type":"ContainerStarted","Data":"d31ca0ee12db673ad0c93767da53c87782d0f81be710285bbf2e25591197c6bd"} Oct 30 17:24:38.421742 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:38.421314 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5477499c6-b5bnw" Oct 30 17:24:38.462915 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:38.462834 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5477499c6-b5bnw" podStartSLOduration=1.9566813220000001 podStartE2EDuration="4.462820545s" totalImagesPullingTime="2.506139223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:38.461483341 +0000 UTC m=+736.513583272" watchObservedRunningTime="2026-10-30 17:24:38.462820545 +0000 UTC m=+736.514920475" Oct 30 17:24:38.657095 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:38.657062 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:38.657419 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:38.657389 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:38.659537 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:38.659510 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 17:24:38.933092 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:38.933056 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-zhw6k"] Oct 30 17:24:38.936591 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:38.936574 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zhw6k" Oct 30 17:24:38.954024 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:38.953999 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-bmbxv\"" Oct 30 17:24:38.954244 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:38.954225 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 17:24:38.955915 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:38.955892 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-zhw6k"] Oct 30 17:24:39.037726 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:39.037690 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-5tdrf"] Oct 30 17:24:39.042149 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:39.042127 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-5tdrf" Oct 30 17:24:39.058198 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:39.058159 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-5tdrf"] Oct 30 17:24:39.062487 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:39.062457 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba865430-c09d-4c91-99f4-183ab4828a2a-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-zhw6k\" (UID: \"ba865430-c09d-4c91-99f4-183ab4828a2a\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zhw6k" Oct 30 17:24:39.062601 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:39.062540 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q68w\" (UniqueName: \"kubernetes.io/projected/ba865430-c09d-4c91-99f4-183ab4828a2a-kube-api-access-7q68w\") pod \"payload-pre-processing-7b98bb678c-zhw6k\" (UID: \"ba865430-c09d-4c91-99f4-183ab4828a2a\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zhw6k" Oct 30 17:24:39.163266 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:39.163231 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d383c0b3-9245-4397-b031-38295a768fba-plugins-config-volume\") pod \"payload-processing-5f78687d5b-5tdrf\" (UID: \"d383c0b3-9245-4397-b031-38295a768fba\") " pod="openshift-ingress/payload-processing-5f78687d5b-5tdrf" Oct 30 17:24:39.163444 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:39.163291 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtszm\" (UniqueName: \"kubernetes.io/projected/d383c0b3-9245-4397-b031-38295a768fba-kube-api-access-wtszm\") pod \"payload-processing-5f78687d5b-5tdrf\" (UID: \"d383c0b3-9245-4397-b031-38295a768fba\") " pod="openshift-ingress/payload-processing-5f78687d5b-5tdrf" Oct 30 17:24:39.163493 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:39.163444 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q68w\" (UniqueName: \"kubernetes.io/projected/ba865430-c09d-4c91-99f4-183ab4828a2a-kube-api-access-7q68w\") pod \"payload-pre-processing-7b98bb678c-zhw6k\" (UID: \"ba865430-c09d-4c91-99f4-183ab4828a2a\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zhw6k" Oct 30 17:24:39.163564 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:39.163544 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba865430-c09d-4c91-99f4-183ab4828a2a-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-zhw6k\" (UID: \"ba865430-c09d-4c91-99f4-183ab4828a2a\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zhw6k" Oct 30 17:24:39.164343 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:39.164320 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba865430-c09d-4c91-99f4-183ab4828a2a-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-zhw6k\" (UID: \"ba865430-c09d-4c91-99f4-183ab4828a2a\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zhw6k" Oct 30 17:24:39.186579 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:39.186508 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q68w\" (UniqueName: \"kubernetes.io/projected/ba865430-c09d-4c91-99f4-183ab4828a2a-kube-api-access-7q68w\") pod \"payload-pre-processing-7b98bb678c-zhw6k\" (UID: \"ba865430-c09d-4c91-99f4-183ab4828a2a\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zhw6k" Oct 30 17:24:39.248959 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:39.248929 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zhw6k" Oct 30 17:24:39.264674 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:39.264641 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d383c0b3-9245-4397-b031-38295a768fba-plugins-config-volume\") pod \"payload-processing-5f78687d5b-5tdrf\" (UID: \"d383c0b3-9245-4397-b031-38295a768fba\") " pod="openshift-ingress/payload-processing-5f78687d5b-5tdrf" Oct 30 17:24:39.264808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:39.264788 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtszm\" (UniqueName: \"kubernetes.io/projected/d383c0b3-9245-4397-b031-38295a768fba-kube-api-access-wtszm\") pod \"payload-processing-5f78687d5b-5tdrf\" (UID: \"d383c0b3-9245-4397-b031-38295a768fba\") " pod="openshift-ingress/payload-processing-5f78687d5b-5tdrf" Oct 30 17:24:39.265356 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:39.265335 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d383c0b3-9245-4397-b031-38295a768fba-plugins-config-volume\") pod \"payload-processing-5f78687d5b-5tdrf\" (UID: \"d383c0b3-9245-4397-b031-38295a768fba\") " pod="openshift-ingress/payload-processing-5f78687d5b-5tdrf" Oct 30 17:24:39.288649 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:39.288612 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtszm\" (UniqueName: \"kubernetes.io/projected/d383c0b3-9245-4397-b031-38295a768fba-kube-api-access-wtszm\") pod \"payload-processing-5f78687d5b-5tdrf\" (UID: \"d383c0b3-9245-4397-b031-38295a768fba\") " pod="openshift-ingress/payload-processing-5f78687d5b-5tdrf" Oct 30 17:24:39.355016 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:39.354974 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-5tdrf" Oct 30 17:24:39.559005 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:39.558982 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-zhw6k"] Oct 30 17:24:39.561032 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:24:39.561007 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba865430_c09d_4c91_99f4_183ab4828a2a.slice/crio-d894e11f267dffda35c6112d0220fd0ce52a3958e749ddc2c1eac8cd6c1b3dc7 WatchSource:0}: Error finding container d894e11f267dffda35c6112d0220fd0ce52a3958e749ddc2c1eac8cd6c1b3dc7: Status 404 returned error can't find the container with id d894e11f267dffda35c6112d0220fd0ce52a3958e749ddc2c1eac8cd6c1b3dc7 Oct 30 17:24:39.669410 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:39.669381 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-5tdrf"] Oct 30 17:24:39.670417 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:24:39.670392 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd383c0b3_9245_4397_b031_38295a768fba.slice/crio-6b07536f4d1d11fc6c99f985172e0428c1b9a16b6c62c129abc40a3a41496f1a WatchSource:0}: Error finding container 6b07536f4d1d11fc6c99f985172e0428c1b9a16b6c62c129abc40a3a41496f1a: Status 404 returned error can't find the container with id 6b07536f4d1d11fc6c99f985172e0428c1b9a16b6c62c129abc40a3a41496f1a Oct 30 17:24:40.434738 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:40.434660 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-5tdrf" event={"ID":"d383c0b3-9245-4397-b031-38295a768fba","Type":"ContainerStarted","Data":"6b07536f4d1d11fc6c99f985172e0428c1b9a16b6c62c129abc40a3a41496f1a"} Oct 30 17:24:40.436362 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:40.436268 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zhw6k" event={"ID":"ba865430-c09d-4c91-99f4-183ab4828a2a","Type":"ContainerStarted","Data":"d894e11f267dffda35c6112d0220fd0ce52a3958e749ddc2c1eac8cd6c1b3dc7"} Oct 30 17:24:41.832965 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:41.832930 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m"] Oct 30 17:24:41.867284 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:41.867247 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m"] Oct 30 17:24:41.867459 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:41.867371 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:41.892147 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:41.892114 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-xmr9m\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:41.892292 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:41.892161 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-xmr9m\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:41.892292 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:41.892195 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-xmr9m\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:41.892292 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:41.892211 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-xmr9m\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:41.892292 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:41.892232 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qlg6\" (UniqueName: \"kubernetes.io/projected/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-kube-api-access-2qlg6\") pod \"maas-default-gateway-openshift-default-7df854bf67-xmr9m\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:41.892292 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:41.892271 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-xmr9m\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:41.892534 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:41.892307 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-xmr9m\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:41.892534 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:41.892334 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-xmr9m\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:41.892534 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:41.892355 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-xmr9m\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:41.993818 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:41.993783 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-xmr9m\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:41.993818 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:41.993827 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-xmr9m\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:41.994104 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:41.993888 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-xmr9m\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:41.994104 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:41.993909 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-xmr9m\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:41.994104 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:41.993942 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qlg6\" (UniqueName: \"kubernetes.io/projected/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-kube-api-access-2qlg6\") pod \"maas-default-gateway-openshift-default-7df854bf67-xmr9m\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:41.994104 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:41.994009 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-xmr9m\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:41.994104 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:41.994074 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-xmr9m\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:41.994104 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:41.994102 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-xmr9m\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:41.994402 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:41.994131 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-xmr9m\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:41.994606 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:41.994574 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-xmr9m\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:41.994764 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:41.994744 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-xmr9m\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:41.995209 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:41.995179 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-xmr9m\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:41.995320 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:41.995300 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-xmr9m\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:41.995516 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:41.995494 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-xmr9m\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:41.996956 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:41.996931 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-xmr9m\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:41.997400 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:41.997381 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-xmr9m\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:42.021830 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:42.021803 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-xmr9m\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:42.022003 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:42.021982 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qlg6\" (UniqueName: \"kubernetes.io/projected/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-kube-api-access-2qlg6\") pod \"maas-default-gateway-openshift-default-7df854bf67-xmr9m\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:42.188776 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:42.188746 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:42.448373 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:42.448287 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-5tdrf" event={"ID":"d383c0b3-9245-4397-b031-38295a768fba","Type":"ContainerStarted","Data":"40c4b23bd782f37011939209c39d1f7369f6242d84f14cfc91489dcb88a51b46"} Oct 30 17:24:42.448522 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:42.448412 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-5tdrf" Oct 30 17:24:42.449855 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:42.449831 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zhw6k" event={"ID":"ba865430-c09d-4c91-99f4-183ab4828a2a","Type":"ContainerStarted","Data":"6a3a199cf42a031c46f43d59954e758f706c005e0605bd69dfdb8599f4edc978"} Oct 30 17:24:42.449957 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:42.449912 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zhw6k" Oct 30 17:24:42.492628 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:42.492577 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-5tdrf" podStartSLOduration=1.041341517 podStartE2EDuration="3.492564355s" totalImagesPullingTime="2.451222838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:42.491370221 +0000 UTC m=+740.543470153" watchObservedRunningTime="2026-10-30 17:24:42.492564355 +0000 UTC m=+740.544664312" Oct 30 17:24:42.508967 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:42.508945 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m"] Oct 30 17:24:42.510460 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:24:42.510438 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd054b35e_8973_4d7b_84e5_6a49ac14bfe1.slice/crio-9c455729f4480efc778983c52e6ded4dc15a4b0388209ef53c10e047cda5188f WatchSource:0}: Error finding container 9c455729f4480efc778983c52e6ded4dc15a4b0388209ef53c10e047cda5188f: Status 404 returned error can't find the container with id 9c455729f4480efc778983c52e6ded4dc15a4b0388209ef53c10e047cda5188f Oct 30 17:24:42.513813 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:42.513786 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:24:42.513890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:42.513847 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:24:42.513927 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:42.513897 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:24:42.532815 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:42.532753 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zhw6k" podStartSLOduration=1.897277655 podStartE2EDuration="4.532735213s" totalImagesPullingTime="2.635457558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:42.531930654 +0000 UTC m=+740.584030586" watchObservedRunningTime="2026-10-30 17:24:42.532735213 +0000 UTC m=+740.584835145" Oct 30 17:24:43.456071 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:43.456035 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" event={"ID":"d054b35e-8973-4d7b-84e5-6a49ac14bfe1","Type":"ContainerStarted","Data":"deb2369a26a05fd0b308a505763c7e9a7a6a472dde1de9fa147fea3f28af0fe9"} Oct 30 17:24:43.456487 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:43.456093 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" event={"ID":"d054b35e-8973-4d7b-84e5-6a49ac14bfe1","Type":"ContainerStarted","Data":"9c455729f4480efc778983c52e6ded4dc15a4b0388209ef53c10e047cda5188f"} Oct 30 17:24:43.518206 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:43.518147 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" podStartSLOduration=2.5181344230000002 podStartE2EDuration="2.518134423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:43.518044986 +0000 UTC m=+741.570144917" watchObservedRunningTime="2026-10-30 17:24:43.518134423 +0000 UTC m=+741.570234353" Oct 30 17:24:44.189488 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:44.189447 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:44.191028 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:44.191005 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-xmr9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 17:24:44.191088 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:44.191053 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" podUID="d054b35e-8973-4d7b-84e5-6a49ac14bfe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 17:24:45.189206 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:45.189173 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-xmr9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 17:24:45.189610 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:45.189238 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" podUID="d054b35e-8973-4d7b-84e5-6a49ac14bfe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 17:24:46.189338 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:46.189303 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-xmr9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 17:24:46.189739 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:46.189359 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" podUID="d054b35e-8973-4d7b-84e5-6a49ac14bfe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 17:24:47.189368 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:47.189331 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-xmr9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 17:24:47.189765 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:47.189384 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" podUID="d054b35e-8973-4d7b-84e5-6a49ac14bfe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 17:24:48.189674 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:48.189642 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-xmr9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 17:24:48.190175 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:48.190141 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" podUID="d054b35e-8973-4d7b-84e5-6a49ac14bfe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 17:24:49.189967 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:49.189934 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-xmr9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 17:24:49.190337 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:49.189985 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" podUID="d054b35e-8973-4d7b-84e5-6a49ac14bfe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 17:24:49.429951 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:49.429911 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5477499c6-b5bnw" Oct 30 17:24:50.789766 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:50.789733 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:50.790327 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:50.790300 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:50.791024 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:50.791003 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:24:53.459075 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:53.459046 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-5tdrf" Oct 30 17:24:53.459466 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:24:53.459120 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zhw6k" Oct 30 17:25:04.197791 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:25:04.197716 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5477499c6-b5bnw"] Oct 30 17:25:04.198190 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:25:04.197967 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5477499c6-b5bnw" podUID="18a610f0-be91-40cb-900e-7e009f041fed" containerName="manager" containerID="cri-o://d31ca0ee12db673ad0c93767da53c87782d0f81be710285bbf2e25591197c6bd" gracePeriod=30 Oct 30 17:25:04.453125 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:25:04.453060 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5477499c6-b5bnw" Oct 30 17:25:04.595406 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:25:04.595366 4965 generic.go:356] "Generic (PLEG): container finished" podID="18a610f0-be91-40cb-900e-7e009f041fed" containerID="d31ca0ee12db673ad0c93767da53c87782d0f81be710285bbf2e25591197c6bd" exitCode=0 Oct 30 17:25:04.595406 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:25:04.595413 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5477499c6-b5bnw" Oct 30 17:25:04.595606 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:25:04.595450 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5477499c6-b5bnw" event={"ID":"18a610f0-be91-40cb-900e-7e009f041fed","Type":"ContainerDied","Data":"d31ca0ee12db673ad0c93767da53c87782d0f81be710285bbf2e25591197c6bd"} Oct 30 17:25:04.595606 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:25:04.595506 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5477499c6-b5bnw" event={"ID":"18a610f0-be91-40cb-900e-7e009f041fed","Type":"ContainerDied","Data":"92d074abe023106b7c7d7fb463bfdae75b1f25545c35a601a1ca87b7d4ca4c22"} Oct 30 17:25:04.595606 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:25:04.595530 4965 scope.go:122] "RemoveContainer" containerID="d31ca0ee12db673ad0c93767da53c87782d0f81be710285bbf2e25591197c6bd" Oct 30 17:25:04.604578 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:25:04.604556 4965 scope.go:122] "RemoveContainer" containerID="d31ca0ee12db673ad0c93767da53c87782d0f81be710285bbf2e25591197c6bd" Oct 30 17:25:04.604847 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:25:04.604825 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d31ca0ee12db673ad0c93767da53c87782d0f81be710285bbf2e25591197c6bd\": container with ID starting with d31ca0ee12db673ad0c93767da53c87782d0f81be710285bbf2e25591197c6bd not found: ID does not exist" containerID="d31ca0ee12db673ad0c93767da53c87782d0f81be710285bbf2e25591197c6bd" Oct 30 17:25:04.604945 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:25:04.604855 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d31ca0ee12db673ad0c93767da53c87782d0f81be710285bbf2e25591197c6bd"} err="failed to get container status \"d31ca0ee12db673ad0c93767da53c87782d0f81be710285bbf2e25591197c6bd\": rpc error: code = NotFound desc = could not find container \"d31ca0ee12db673ad0c93767da53c87782d0f81be710285bbf2e25591197c6bd\": container with ID starting with d31ca0ee12db673ad0c93767da53c87782d0f81be710285bbf2e25591197c6bd not found: ID does not exist" Oct 30 17:25:04.608132 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:25:04.608114 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18a610f0-be91-40cb-900e-7e009f041fed-kube-api-access-tvkcv\" (UniqueName: \"kubernetes.io/projected/18a610f0-be91-40cb-900e-7e009f041fed-kube-api-access-tvkcv\") pod \"18a610f0-be91-40cb-900e-7e009f041fed\" (UID: \"18a610f0-be91-40cb-900e-7e009f041fed\") " Oct 30 17:25:04.608323 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:25:04.608305 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18a610f0-be91-40cb-900e-7e009f041fed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/18a610f0-be91-40cb-900e-7e009f041fed-metrics-certs\") pod \"18a610f0-be91-40cb-900e-7e009f041fed\" (UID: \"18a610f0-be91-40cb-900e-7e009f041fed\") " Oct 30 17:25:04.610333 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:25:04.610289 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a610f0-be91-40cb-900e-7e009f041fed-kube-api-access-tvkcv" pod "18a610f0-be91-40cb-900e-7e009f041fed" (UID: "18a610f0-be91-40cb-900e-7e009f041fed"). InnerVolumeSpecName "kube-api-access-tvkcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:04.610430 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:25:04.610385 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18a610f0-be91-40cb-900e-7e009f041fed-metrics-certs" pod "18a610f0-be91-40cb-900e-7e009f041fed" (UID: "18a610f0-be91-40cb-900e-7e009f041fed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:25:04.709323 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:25:04.709241 4965 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18a610f0-be91-40cb-900e-7e009f041fed-metrics-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:25:04.709323 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:25:04.709269 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvkcv\" (UniqueName: \"kubernetes.io/projected/18a610f0-be91-40cb-900e-7e009f041fed-kube-api-access-tvkcv\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:25:04.942333 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:25:04.942294 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5477499c6-b5bnw"] Oct 30 17:25:04.954548 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:25:04.954516 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5477499c6-b5bnw"] Oct 30 17:25:05.995857 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:25:05.995822 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:25:06.096393 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:25:06.096359 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18a610f0-be91-40cb-900e-7e009f041fed" path="/var/lib/kubelet/pods/18a610f0-be91-40cb-900e-7e009f041fed/volumes" Oct 30 17:25:35.995677 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:25:35.995640 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:26:05.995513 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:26:05.995469 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:26:22.026917 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.026847 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2"] Oct 30 17:26:22.027554 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.027535 4965 cpu_manager.go:414] "containerMap: removing container" podUID="18a610f0-be91-40cb-900e-7e009f041fed" containerName="manager" Oct 30 17:26:22.027639 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.027555 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18a610f0-be91-40cb-900e-7e009f041fed" containerName="manager" Oct 30 17:26:22.027674 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.027666 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="18a610f0-be91-40cb-900e-7e009f041fed" containerName="manager" Oct 30 17:26:22.031347 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.031319 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:22.046671 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.046639 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-db98b0-openshift-default-dockercfg-qvj6k\"" Oct 30 17:26:22.060542 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.060516 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2"] Oct 30 17:26:22.117988 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.117959 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-istio-envoy\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:22.118144 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.118007 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-credential-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:22.118144 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.118095 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232650b5-3cab-4611-8ae6-9504a7ac2f04-istio-podinfo\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:22.118144 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.118129 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhb8r\" (UniqueName: \"kubernetes.io/projected/232650b5-3cab-4611-8ae6-9504a7ac2f04-kube-api-access-vhb8r\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:22.118261 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.118147 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232650b5-3cab-4611-8ae6-9504a7ac2f04-istio-token\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:22.118294 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.118256 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-workload-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:22.118294 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.118276 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-workload-certs\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:22.118359 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.118294 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-istio-data\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:22.118359 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.118324 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232650b5-3cab-4611-8ae6-9504a7ac2f04-istiod-ca-cert\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:22.219339 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.219308 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-istio-envoy\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:22.219528 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.219362 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-credential-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:22.219528 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.219397 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232650b5-3cab-4611-8ae6-9504a7ac2f04-istio-podinfo\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:22.219528 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.219421 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhb8r\" (UniqueName: \"kubernetes.io/projected/232650b5-3cab-4611-8ae6-9504a7ac2f04-kube-api-access-vhb8r\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:22.219528 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.219439 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232650b5-3cab-4611-8ae6-9504a7ac2f04-istio-token\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:22.219740 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.219658 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-workload-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:22.219740 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.219691 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-workload-certs\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:22.219850 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.219738 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-istio-data\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:22.219850 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.219821 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232650b5-3cab-4611-8ae6-9504a7ac2f04-istiod-ca-cert\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:22.220089 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.219955 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-credential-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:22.220147 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.220081 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-workload-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:22.220147 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.220116 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-workload-certs\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:22.220319 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.220296 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-istio-data\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:22.220547 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.220529 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232650b5-3cab-4611-8ae6-9504a7ac2f04-istiod-ca-cert\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:22.221639 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.221616 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-istio-envoy\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:22.221977 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.221960 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232650b5-3cab-4611-8ae6-9504a7ac2f04-istio-podinfo\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:22.243047 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.243025 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232650b5-3cab-4611-8ae6-9504a7ac2f04-istio-token\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:22.243157 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.243135 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhb8r\" (UniqueName: \"kubernetes.io/projected/232650b5-3cab-4611-8ae6-9504a7ac2f04-kube-api-access-vhb8r\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:22.314809 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.314739 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt"] Oct 30 17:26:22.320614 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.320590 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:22.350296 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.350269 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt"] Oct 30 17:26:22.359037 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.359012 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-db98b0-openshift-default-dockercfg-qvj6k\"" Oct 30 17:26:22.365491 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.365471 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:22.388824 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.388802 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-0e7403-openshift-default-dockercfg-z9pr5\"" Oct 30 17:26:22.422669 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.422450 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istio-token\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:22.422669 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.422521 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istio-envoy\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:22.422669 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.422590 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-workload-certs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:22.422669 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.422644 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istio-podinfo\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:22.422948 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.422673 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istiod-ca-cert\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:22.422948 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.422709 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-workload-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:22.422948 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.422756 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-credential-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:22.422948 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.422814 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istio-data\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:22.422948 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.422846 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v82mz\" (UniqueName: \"kubernetes.io/projected/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-kube-api-access-v82mz\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:22.523498 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.523465 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-workload-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:22.523686 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.523505 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-credential-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:22.523686 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.523633 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istio-data\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:22.523686 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.523676 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v82mz\" (UniqueName: \"kubernetes.io/projected/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-kube-api-access-v82mz\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:22.523838 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.523720 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istio-token\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:22.523838 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.523772 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istio-envoy\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:22.523968 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.523841 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-workload-certs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:22.523968 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.523922 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istio-podinfo\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:22.523968 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.523951 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istiod-ca-cert\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:22.524114 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.523964 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-workload-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:22.524114 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.524021 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istio-data\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:22.524318 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.524298 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-workload-certs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:22.524402 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.524381 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-credential-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:22.524659 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.524607 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istiod-ca-cert\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:22.526112 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.526094 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istio-envoy\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:22.526390 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.526371 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istio-podinfo\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:22.547879 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.547841 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istio-token\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:22.548004 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.547943 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v82mz\" (UniqueName: \"kubernetes.io/projected/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-kube-api-access-v82mz\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:22.593666 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.593634 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4"] Oct 30 17:26:22.601512 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.601486 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:22.622347 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.622279 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4"] Oct 30 17:26:22.638969 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.638938 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:22.661546 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.661515 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-506124-openshift-default-dockercfg-xdvhh\"" Oct 30 17:26:22.727570 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.727534 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-workload-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:22.727720 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.727592 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-workload-certs\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:22.727720 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.727622 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istio-envoy\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:22.727720 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.727657 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istio-data\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:22.727720 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.727694 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqtpf\" (UniqueName: \"kubernetes.io/projected/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-kube-api-access-nqtpf\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:22.727877 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.727722 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istio-token\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:22.727877 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.727771 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istio-podinfo\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:22.727877 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.727807 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-credential-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:22.727877 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.727844 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2"] Oct 30 17:26:22.728002 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.727878 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istiod-ca-cert\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:22.733300 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:26:22.733256 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232650b5_3cab_4611_8ae6_9504a7ac2f04.slice/crio-9a7eacddcfc2c9d099e4401ceea0ee9035b3edcd1e514428922bf6a76e12bc20 WatchSource:0}: Error finding container 9a7eacddcfc2c9d099e4401ceea0ee9035b3edcd1e514428922bf6a76e12bc20: Status 404 returned error can't find the container with id 9a7eacddcfc2c9d099e4401ceea0ee9035b3edcd1e514428922bf6a76e12bc20 Oct 30 17:26:22.735747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.735715 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:26:22.735804 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.735779 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:26:22.735848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.735811 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:26:22.833147 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.831341 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istio-data\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:22.833147 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.831410 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqtpf\" (UniqueName: \"kubernetes.io/projected/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-kube-api-access-nqtpf\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:22.833147 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.831445 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istio-token\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:22.833147 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.831504 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istio-podinfo\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:22.833147 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.831545 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-credential-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:22.833147 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.831606 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istiod-ca-cert\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:22.833147 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.831679 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-workload-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:22.833147 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.831724 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-workload-certs\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:22.833147 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.831754 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istio-envoy\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:22.834276 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.834178 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-credential-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:22.834387 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.834332 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istio-data\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:22.834803 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.834580 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-workload-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:22.835029 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.834991 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-workload-certs\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:22.836916 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.836820 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istiod-ca-cert\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:22.839010 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.838960 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istio-envoy\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:22.841241 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.841218 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istio-podinfo\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:22.857997 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.857926 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istio-token\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:22.858150 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.858129 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqtpf\" (UniqueName: \"kubernetes.io/projected/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-kube-api-access-nqtpf\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:22.919196 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.919160 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:22.943512 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.943482 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt"] Oct 30 17:26:22.944945 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:26:22.944919 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d015c9_b3d7_45ae_a4f7_04aa68f678fd.slice/crio-0fb408d1a34b0140bd5497dc6ecdc1833637751b00c421e6a6c18c5354bed4dc WatchSource:0}: Error finding container 0fb408d1a34b0140bd5497dc6ecdc1833637751b00c421e6a6c18c5354bed4dc: Status 404 returned error can't find the container with id 0fb408d1a34b0140bd5497dc6ecdc1833637751b00c421e6a6c18c5354bed4dc Oct 30 17:26:22.947676 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.947635 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:26:22.947796 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.947719 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:26:22.947796 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.947763 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:26:22.962050 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.959984 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" event={"ID":"16d015c9-b3d7-45ae-a4f7-04aa68f678fd","Type":"ContainerStarted","Data":"0fb408d1a34b0140bd5497dc6ecdc1833637751b00c421e6a6c18c5354bed4dc"} Oct 30 17:26:22.962902 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.962803 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" event={"ID":"232650b5-3cab-4611-8ae6-9504a7ac2f04","Type":"ContainerStarted","Data":"df56035585263205e669e35fc96fa35e9dfd07c82318038fc3d0bc5eabc0cdac"} Oct 30 17:26:22.962902 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:22.962853 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" event={"ID":"232650b5-3cab-4611-8ae6-9504a7ac2f04","Type":"ContainerStarted","Data":"9a7eacddcfc2c9d099e4401ceea0ee9035b3edcd1e514428922bf6a76e12bc20"} Oct 30 17:26:23.009932 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:23.009278 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" podStartSLOduration=2.009258966 podStartE2EDuration="2.009258966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:23.007402268 +0000 UTC m=+841.059502224" watchObservedRunningTime="2026-10-30 17:26:23.009258966 +0000 UTC m=+841.061358899" Oct 30 17:26:23.239922 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:23.239898 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4"] Oct 30 17:26:23.241967 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:26:23.241940 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb82e7c_5038_46b7_937e_c9ae4b12b2c0.slice/crio-4db416214e7e09eeb42c38fe4081ef2108444f8e2bf22fa962dfbbe96533702a WatchSource:0}: Error finding container 4db416214e7e09eeb42c38fe4081ef2108444f8e2bf22fa962dfbbe96533702a: Status 404 returned error can't find the container with id 4db416214e7e09eeb42c38fe4081ef2108444f8e2bf22fa962dfbbe96533702a Oct 30 17:26:23.244168 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:23.244139 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:26:23.244256 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:23.244201 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:26:23.244256 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:23.244230 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:26:23.365821 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:23.365741 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:23.373725 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:23.373694 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:23.969547 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:23.969502 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" event={"ID":"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0","Type":"ContainerStarted","Data":"88b179d4d49aef2174d85adf10dae147bf57f623bdfb217c8831e1460c2fe2c2"} Oct 30 17:26:23.969758 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:23.969566 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" event={"ID":"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0","Type":"ContainerStarted","Data":"4db416214e7e09eeb42c38fe4081ef2108444f8e2bf22fa962dfbbe96533702a"} Oct 30 17:26:23.971125 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:23.971098 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" event={"ID":"16d015c9-b3d7-45ae-a4f7-04aa68f678fd","Type":"ContainerStarted","Data":"3756862eda02255279216d61f3e436bd949ab882cb498a8403ebde16a3168476"} Oct 30 17:26:23.971236 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:23.971225 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:23.972110 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:23.972092 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:24.016134 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:24.016084 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" podStartSLOduration=2.016070394 podStartE2EDuration="2.016070394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:24.014374824 +0000 UTC m=+842.066474780" watchObservedRunningTime="2026-10-30 17:26:24.016070394 +0000 UTC m=+842.068170325" Oct 30 17:26:24.060967 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:24.060908 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" podStartSLOduration=2.060883748 podStartE2EDuration="2.060883748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:24.060360481 +0000 UTC m=+842.112460423" watchObservedRunningTime="2026-10-30 17:26:24.060883748 +0000 UTC m=+842.112983682" Oct 30 17:26:24.639241 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:24.639210 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:24.644009 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:24.643984 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:24.920303 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:24.920221 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:24.925927 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:24.925899 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:24.976747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:24.976714 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:24.976955 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:24.976781 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:24.977596 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:24.977568 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:24.977739 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:24.977725 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:26:27.291578 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:27.291545 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w"] Oct 30 17:26:27.295409 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:27.295392 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w" Oct 30 17:26:27.317057 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:27.317024 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-db98b0-serving-cert\"" Oct 30 17:26:27.333368 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:27.332339 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w"] Oct 30 17:26:27.380784 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:27.380747 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4477dfb1-111b-48c7-86ec-2a048339b95c-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w\" (UID: \"4477dfb1-111b-48c7-86ec-2a048339b95c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w" Oct 30 17:26:27.380978 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:27.380799 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4477dfb1-111b-48c7-86ec-2a048339b95c-maas-api-tls\") pod \"maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w\" (UID: \"4477dfb1-111b-48c7-86ec-2a048339b95c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w" Oct 30 17:26:27.380978 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:27.380888 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4477dfb1-111b-48c7-86ec-2a048339b95c-metrics-certs\") pod \"maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w\" (UID: \"4477dfb1-111b-48c7-86ec-2a048339b95c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w" Oct 30 17:26:27.381102 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:27.381012 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg7cb\" (UniqueName: \"kubernetes.io/projected/4477dfb1-111b-48c7-86ec-2a048339b95c-kube-api-access-zg7cb\") pod \"maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w\" (UID: \"4477dfb1-111b-48c7-86ec-2a048339b95c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w" Oct 30 17:26:27.481938 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:27.481906 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4477dfb1-111b-48c7-86ec-2a048339b95c-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w\" (UID: \"4477dfb1-111b-48c7-86ec-2a048339b95c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w" Oct 30 17:26:27.482119 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:27.481946 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4477dfb1-111b-48c7-86ec-2a048339b95c-maas-api-tls\") pod \"maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w\" (UID: \"4477dfb1-111b-48c7-86ec-2a048339b95c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w" Oct 30 17:26:27.482119 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:27.481968 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4477dfb1-111b-48c7-86ec-2a048339b95c-metrics-certs\") pod \"maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w\" (UID: \"4477dfb1-111b-48c7-86ec-2a048339b95c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w" Oct 30 17:26:27.482119 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:27.482021 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg7cb\" (UniqueName: \"kubernetes.io/projected/4477dfb1-111b-48c7-86ec-2a048339b95c-kube-api-access-zg7cb\") pod \"maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w\" (UID: \"4477dfb1-111b-48c7-86ec-2a048339b95c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w" Oct 30 17:26:27.482615 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:27.482593 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4477dfb1-111b-48c7-86ec-2a048339b95c-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w\" (UID: \"4477dfb1-111b-48c7-86ec-2a048339b95c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w" Oct 30 17:26:27.484440 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:27.484418 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4477dfb1-111b-48c7-86ec-2a048339b95c-maas-api-tls\") pod \"maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w\" (UID: \"4477dfb1-111b-48c7-86ec-2a048339b95c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w" Oct 30 17:26:27.484510 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:27.484447 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4477dfb1-111b-48c7-86ec-2a048339b95c-metrics-certs\") pod \"maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w\" (UID: \"4477dfb1-111b-48c7-86ec-2a048339b95c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w" Oct 30 17:26:27.523817 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:27.523772 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg7cb\" (UniqueName: \"kubernetes.io/projected/4477dfb1-111b-48c7-86ec-2a048339b95c-kube-api-access-zg7cb\") pod \"maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w\" (UID: \"4477dfb1-111b-48c7-86ec-2a048339b95c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w" Oct 30 17:26:27.636056 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:27.636024 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w" Oct 30 17:26:27.946494 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:27.946469 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w"] Oct 30 17:26:27.949050 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:26:27.949024 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4477dfb1_111b_48c7_86ec_2a048339b95c.slice/crio-553901b8c2de5cc03e36a81ec2bfa52d9df7ccb65f23abbc2c14eb721d223b6b WatchSource:0}: Error finding container 553901b8c2de5cc03e36a81ec2bfa52d9df7ccb65f23abbc2c14eb721d223b6b: Status 404 returned error can't find the container with id 553901b8c2de5cc03e36a81ec2bfa52d9df7ccb65f23abbc2c14eb721d223b6b Oct 30 17:26:27.991845 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:27.991812 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w" event={"ID":"4477dfb1-111b-48c7-86ec-2a048339b95c","Type":"ContainerStarted","Data":"553901b8c2de5cc03e36a81ec2bfa52d9df7ccb65f23abbc2c14eb721d223b6b"} Oct 30 17:26:28.998544 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:28.998511 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w" event={"ID":"4477dfb1-111b-48c7-86ec-2a048339b95c","Type":"ContainerStarted","Data":"8e444329a7a1fdd59aa81889bdad2734899331284ec8cfd24365988b03bb1742"} Oct 30 17:26:28.998951 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:28.998637 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w" Oct 30 17:26:29.041417 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:29.041365 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w" podStartSLOduration=1.66084036 podStartE2EDuration="2.041351601s" totalImagesPullingTime="380.511241ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:29.040897003 +0000 UTC m=+847.092996931" watchObservedRunningTime="2026-10-30 17:26:29.041351601 +0000 UTC m=+847.093451560" Oct 30 17:26:29.398279 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:29.398249 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-27567a-7c6f9f9747-hhm7h"] Oct 30 17:26:29.402109 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:29.402090 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-27567a-7c6f9f9747-hhm7h" Oct 30 17:26:29.419816 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:29.419785 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-27567a-7c6f9f9747-hhm7h"] Oct 30 17:26:29.473307 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:29.473264 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-27567a\"" Oct 30 17:26:29.473528 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:29.473385 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-27567a-dockercfg-zs465\"" Oct 30 17:26:29.507579 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:29.507542 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-27567a-7c6f9f9747-hhm7h\" (UID: \"9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-27567a-7c6f9f9747-hhm7h" Oct 30 17:26:29.507722 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:29.507598 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx2c5\" (UniqueName: \"kubernetes.io/projected/9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d-kube-api-access-gx2c5\") pod \"payload-pre-processing-e2e-worker-w0-27567a-7c6f9f9747-hhm7h\" (UID: \"9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-27567a-7c6f9f9747-hhm7h" Oct 30 17:26:29.609306 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:29.609267 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-27567a-7c6f9f9747-hhm7h\" (UID: \"9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-27567a-7c6f9f9747-hhm7h" Oct 30 17:26:29.609491 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:29.609320 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx2c5\" (UniqueName: \"kubernetes.io/projected/9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d-kube-api-access-gx2c5\") pod \"payload-pre-processing-e2e-worker-w0-27567a-7c6f9f9747-hhm7h\" (UID: \"9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-27567a-7c6f9f9747-hhm7h" Oct 30 17:26:29.610049 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:29.610030 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-27567a-7c6f9f9747-hhm7h\" (UID: \"9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-27567a-7c6f9f9747-hhm7h" Oct 30 17:26:29.633685 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:29.633624 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx2c5\" (UniqueName: \"kubernetes.io/projected/9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d-kube-api-access-gx2c5\") pod \"payload-pre-processing-e2e-worker-w0-27567a-7c6f9f9747-hhm7h\" (UID: \"9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-27567a-7c6f9f9747-hhm7h" Oct 30 17:26:29.715802 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:29.715726 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-27567a-7c6f9f9747-hhm7h" Oct 30 17:26:30.014332 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:30.014305 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-27567a-7c6f9f9747-hhm7h"] Oct 30 17:26:30.015964 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:26:30.015937 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a4c9a42_ca45_4f5b_96c3_6b3bad58f62d.slice/crio-4745739e47ab05dd18b31ab0e14cc22c45cbd8b85720a1d0e19cc5110961c9d4 WatchSource:0}: Error finding container 4745739e47ab05dd18b31ab0e14cc22c45cbd8b85720a1d0e19cc5110961c9d4: Status 404 returned error can't find the container with id 4745739e47ab05dd18b31ab0e14cc22c45cbd8b85720a1d0e19cc5110961c9d4 Oct 30 17:26:30.294850 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:30.294760 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-455086-6c947b47-fllqz"] Oct 30 17:26:30.298828 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:30.298804 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-455086-6c947b47-fllqz" Oct 30 17:26:30.319007 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:30.318968 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-455086-6c947b47-fllqz"] Oct 30 17:26:30.379280 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:30.371157 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-455086-dockercfg-7lv9w\"" Oct 30 17:26:30.379280 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:30.371672 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-455086\"" Oct 30 17:26:30.418191 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:30.418138 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfcedd36-934a-4257-b1e9-db368ed2bcd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-455086-6c947b47-fllqz\" (UID: \"dfcedd36-934a-4257-b1e9-db368ed2bcd5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-455086-6c947b47-fllqz" Oct 30 17:26:30.418367 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:30.418263 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jlrn\" (UniqueName: \"kubernetes.io/projected/dfcedd36-934a-4257-b1e9-db368ed2bcd5-kube-api-access-8jlrn\") pod \"payload-pre-processing-e2e-worker-w1-455086-6c947b47-fllqz\" (UID: \"dfcedd36-934a-4257-b1e9-db368ed2bcd5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-455086-6c947b47-fllqz" Oct 30 17:26:30.518967 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:30.518932 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfcedd36-934a-4257-b1e9-db368ed2bcd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-455086-6c947b47-fllqz\" (UID: \"dfcedd36-934a-4257-b1e9-db368ed2bcd5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-455086-6c947b47-fllqz" Oct 30 17:26:30.519152 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:30.518990 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jlrn\" (UniqueName: \"kubernetes.io/projected/dfcedd36-934a-4257-b1e9-db368ed2bcd5-kube-api-access-8jlrn\") pod \"payload-pre-processing-e2e-worker-w1-455086-6c947b47-fllqz\" (UID: \"dfcedd36-934a-4257-b1e9-db368ed2bcd5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-455086-6c947b47-fllqz" Oct 30 17:26:30.519689 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:30.519664 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfcedd36-934a-4257-b1e9-db368ed2bcd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-455086-6c947b47-fllqz\" (UID: \"dfcedd36-934a-4257-b1e9-db368ed2bcd5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-455086-6c947b47-fllqz" Oct 30 17:26:30.542227 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:30.542198 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jlrn\" (UniqueName: \"kubernetes.io/projected/dfcedd36-934a-4257-b1e9-db368ed2bcd5-kube-api-access-8jlrn\") pod \"payload-pre-processing-e2e-worker-w1-455086-6c947b47-fllqz\" (UID: \"dfcedd36-934a-4257-b1e9-db368ed2bcd5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-455086-6c947b47-fllqz" Oct 30 17:26:30.612489 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:30.612460 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-455086-6c947b47-fllqz" Oct 30 17:26:30.916089 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:30.916059 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-455086-6c947b47-fllqz"] Oct 30 17:26:30.918132 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:26:30.918105 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfcedd36_934a_4257_b1e9_db368ed2bcd5.slice/crio-72422bec88f7b802e9b33dc5d647f0281af5eb05c420b887466c4810318ec103 WatchSource:0}: Error finding container 72422bec88f7b802e9b33dc5d647f0281af5eb05c420b887466c4810318ec103: Status 404 returned error can't find the container with id 72422bec88f7b802e9b33dc5d647f0281af5eb05c420b887466c4810318ec103 Oct 30 17:26:31.010551 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:31.010519 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-27567a-7c6f9f9747-hhm7h" event={"ID":"9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d","Type":"ContainerStarted","Data":"3ba69e2ce57bf5f1e104e01f625eb17e39014f25b3c2f7b5276f13d42a0a54a3"} Oct 30 17:26:31.010713 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:31.010694 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-27567a-7c6f9f9747-hhm7h" Oct 30 17:26:31.010801 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:31.010736 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-27567a-7c6f9f9747-hhm7h" event={"ID":"9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d","Type":"ContainerStarted","Data":"4745739e47ab05dd18b31ab0e14cc22c45cbd8b85720a1d0e19cc5110961c9d4"} Oct 30 17:26:31.012098 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:31.012081 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-455086-6c947b47-fllqz" event={"ID":"dfcedd36-934a-4257-b1e9-db368ed2bcd5","Type":"ContainerStarted","Data":"1a95329f71d22d5c904d568dae3ca624bb25ed85e95cf68a0ed5eba499cadb72"} Oct 30 17:26:31.012201 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:31.012114 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-455086-6c947b47-fllqz" event={"ID":"dfcedd36-934a-4257-b1e9-db368ed2bcd5","Type":"ContainerStarted","Data":"72422bec88f7b802e9b33dc5d647f0281af5eb05c420b887466c4810318ec103"} Oct 30 17:26:31.012201 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:31.012134 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-455086-6c947b47-fllqz" Oct 30 17:26:31.050260 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:31.050215 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-27567a-7c6f9f9747-hhm7h" podStartSLOduration=2.050200124 podStartE2EDuration="2.050200124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:31.050010564 +0000 UTC m=+849.102110514" watchObservedRunningTime="2026-10-30 17:26:31.050200124 +0000 UTC m=+849.102300055" Oct 30 17:26:31.091941 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:31.091899 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-455086-6c947b47-fllqz" podStartSLOduration=1.091885578 podStartE2EDuration="1.091885578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:31.090267691 +0000 UTC m=+849.142367698" watchObservedRunningTime="2026-10-30 17:26:31.091885578 +0000 UTC m=+849.143985515" Oct 30 17:26:31.326069 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:31.326035 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:26:32.679888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:32.677629 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-506124-59bc6cd6c4-rp7pn"] Oct 30 17:26:32.682776 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:32.682751 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-506124-59bc6cd6c4-rp7pn" Oct 30 17:26:32.701342 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:32.701308 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-506124-59bc6cd6c4-rp7pn"] Oct 30 17:26:32.710878 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:32.710834 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-506124\"" Oct 30 17:26:32.711014 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:32.710911 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-506124-dockercfg-rck28\"" Oct 30 17:26:32.750977 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:32.750949 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4scf\" (UniqueName: \"kubernetes.io/projected/6540385c-e0cf-41cd-8e45-123e600e2038-kube-api-access-b4scf\") pod \"payload-processing-e2e-worker-w2-506124-59bc6cd6c4-rp7pn\" (UID: \"6540385c-e0cf-41cd-8e45-123e600e2038\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-506124-59bc6cd6c4-rp7pn" Oct 30 17:26:32.751125 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:32.751004 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6540385c-e0cf-41cd-8e45-123e600e2038-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-506124-59bc6cd6c4-rp7pn\" (UID: \"6540385c-e0cf-41cd-8e45-123e600e2038\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-506124-59bc6cd6c4-rp7pn" Oct 30 17:26:32.851736 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:32.851702 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4scf\" (UniqueName: \"kubernetes.io/projected/6540385c-e0cf-41cd-8e45-123e600e2038-kube-api-access-b4scf\") pod \"payload-processing-e2e-worker-w2-506124-59bc6cd6c4-rp7pn\" (UID: \"6540385c-e0cf-41cd-8e45-123e600e2038\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-506124-59bc6cd6c4-rp7pn" Oct 30 17:26:32.851921 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:32.851753 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6540385c-e0cf-41cd-8e45-123e600e2038-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-506124-59bc6cd6c4-rp7pn\" (UID: \"6540385c-e0cf-41cd-8e45-123e600e2038\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-506124-59bc6cd6c4-rp7pn" Oct 30 17:26:32.852570 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:32.852550 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6540385c-e0cf-41cd-8e45-123e600e2038-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-506124-59bc6cd6c4-rp7pn\" (UID: \"6540385c-e0cf-41cd-8e45-123e600e2038\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-506124-59bc6cd6c4-rp7pn" Oct 30 17:26:32.883558 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:32.883523 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4scf\" (UniqueName: \"kubernetes.io/projected/6540385c-e0cf-41cd-8e45-123e600e2038-kube-api-access-b4scf\") pod \"payload-processing-e2e-worker-w2-506124-59bc6cd6c4-rp7pn\" (UID: \"6540385c-e0cf-41cd-8e45-123e600e2038\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-506124-59bc6cd6c4-rp7pn" Oct 30 17:26:32.996684 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:32.996573 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-506124-59bc6cd6c4-rp7pn" Oct 30 17:26:33.046605 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:33.046575 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm"] Oct 30 17:26:33.053559 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:33.052730 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm" Oct 30 17:26:33.071028 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:33.069151 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-27567a-serving-cert\"" Oct 30 17:26:33.076821 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:33.072797 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm"] Oct 30 17:26:33.154804 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:33.154772 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jn2x\" (UniqueName: \"kubernetes.io/projected/d9665073-6722-4274-bc2d-d453b06bd6c0-kube-api-access-4jn2x\") pod \"maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm\" (UID: \"d9665073-6722-4274-bc2d-d453b06bd6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm" Oct 30 17:26:33.154804 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:33.154808 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9665073-6722-4274-bc2d-d453b06bd6c0-metrics-certs\") pod \"maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm\" (UID: \"d9665073-6722-4274-bc2d-d453b06bd6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm" Oct 30 17:26:33.155046 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:33.154994 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9665073-6722-4274-bc2d-d453b06bd6c0-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm\" (UID: \"d9665073-6722-4274-bc2d-d453b06bd6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm" Oct 30 17:26:33.155110 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:33.155085 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9665073-6722-4274-bc2d-d453b06bd6c0-maas-api-tls\") pod \"maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm\" (UID: \"d9665073-6722-4274-bc2d-d453b06bd6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm" Oct 30 17:26:33.256235 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:33.256148 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9665073-6722-4274-bc2d-d453b06bd6c0-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm\" (UID: \"d9665073-6722-4274-bc2d-d453b06bd6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm" Oct 30 17:26:33.256235 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:33.256208 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9665073-6722-4274-bc2d-d453b06bd6c0-maas-api-tls\") pod \"maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm\" (UID: \"d9665073-6722-4274-bc2d-d453b06bd6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm" Oct 30 17:26:33.256531 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:33.256256 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jn2x\" (UniqueName: \"kubernetes.io/projected/d9665073-6722-4274-bc2d-d453b06bd6c0-kube-api-access-4jn2x\") pod \"maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm\" (UID: \"d9665073-6722-4274-bc2d-d453b06bd6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm" Oct 30 17:26:33.256531 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:33.256278 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9665073-6722-4274-bc2d-d453b06bd6c0-metrics-certs\") pod \"maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm\" (UID: \"d9665073-6722-4274-bc2d-d453b06bd6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm" Oct 30 17:26:33.282887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:33.257006 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9665073-6722-4274-bc2d-d453b06bd6c0-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm\" (UID: \"d9665073-6722-4274-bc2d-d453b06bd6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm" Oct 30 17:26:33.282887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:33.279489 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9665073-6722-4274-bc2d-d453b06bd6c0-maas-api-tls\") pod \"maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm\" (UID: \"d9665073-6722-4274-bc2d-d453b06bd6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm" Oct 30 17:26:33.283698 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:33.283669 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9665073-6722-4274-bc2d-d453b06bd6c0-metrics-certs\") pod \"maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm\" (UID: \"d9665073-6722-4274-bc2d-d453b06bd6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm" Oct 30 17:26:33.291243 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:33.291220 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jn2x\" (UniqueName: \"kubernetes.io/projected/d9665073-6722-4274-bc2d-d453b06bd6c0-kube-api-access-4jn2x\") pod \"maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm\" (UID: \"d9665073-6722-4274-bc2d-d453b06bd6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm" Oct 30 17:26:33.301944 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:33.301922 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-506124-59bc6cd6c4-rp7pn"] Oct 30 17:26:33.303642 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:26:33.303613 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6540385c_e0cf_41cd_8e45_123e600e2038.slice/crio-de64fa2a230f99f0974c77773d3fb532be8bb76acef2d9a82cdd024a8aeea329 WatchSource:0}: Error finding container de64fa2a230f99f0974c77773d3fb532be8bb76acef2d9a82cdd024a8aeea329: Status 404 returned error can't find the container with id de64fa2a230f99f0974c77773d3fb532be8bb76acef2d9a82cdd024a8aeea329 Oct 30 17:26:33.413549 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:33.413522 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm" Oct 30 17:26:33.734506 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:33.734480 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm"] Oct 30 17:26:33.738924 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:26:33.737227 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9665073_6722_4274_bc2d_d453b06bd6c0.slice/crio-395323d4fb41dba57c1727e9a7e664eafbfdaec930f31328ae8c9cfba69dcc9c WatchSource:0}: Error finding container 395323d4fb41dba57c1727e9a7e664eafbfdaec930f31328ae8c9cfba69dcc9c: Status 404 returned error can't find the container with id 395323d4fb41dba57c1727e9a7e664eafbfdaec930f31328ae8c9cfba69dcc9c Oct 30 17:26:34.031449 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:34.031362 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-506124-59bc6cd6c4-rp7pn" event={"ID":"6540385c-e0cf-41cd-8e45-123e600e2038","Type":"ContainerStarted","Data":"4db60a6605f95d02333f5748f3e1dbb409bd4cfd1cea30df1069105acaff950f"} Oct 30 17:26:34.031449 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:34.031415 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-506124-59bc6cd6c4-rp7pn" event={"ID":"6540385c-e0cf-41cd-8e45-123e600e2038","Type":"ContainerStarted","Data":"de64fa2a230f99f0974c77773d3fb532be8bb76acef2d9a82cdd024a8aeea329"} Oct 30 17:26:34.031709 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:34.031473 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-506124-59bc6cd6c4-rp7pn" Oct 30 17:26:34.032687 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:34.032655 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm" event={"ID":"d9665073-6722-4274-bc2d-d453b06bd6c0","Type":"ContainerStarted","Data":"395323d4fb41dba57c1727e9a7e664eafbfdaec930f31328ae8c9cfba69dcc9c"} Oct 30 17:26:34.077977 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:34.077906 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-506124-59bc6cd6c4-rp7pn" podStartSLOduration=2.077887748 podStartE2EDuration="2.077887748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:34.077229128 +0000 UTC m=+852.129329077" watchObservedRunningTime="2026-10-30 17:26:34.077887748 +0000 UTC m=+852.129987677" Oct 30 17:26:34.774888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:34.770555 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46"] Oct 30 17:26:34.786094 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:34.786063 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46" Oct 30 17:26:34.793995 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:34.793969 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46"] Oct 30 17:26:34.801577 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:34.801555 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-455086-serving-cert\"" Oct 30 17:26:34.896667 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:34.896614 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67332f71-74ce-4486-a341-cd32c7a3a9ef-metrics-certs\") pod \"maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46\" (UID: \"67332f71-74ce-4486-a341-cd32c7a3a9ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46" Oct 30 17:26:34.896856 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:34.896681 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67332f71-74ce-4486-a341-cd32c7a3a9ef-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46\" (UID: \"67332f71-74ce-4486-a341-cd32c7a3a9ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46" Oct 30 17:26:34.896856 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:34.896770 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lblv\" (UniqueName: \"kubernetes.io/projected/67332f71-74ce-4486-a341-cd32c7a3a9ef-kube-api-access-2lblv\") pod \"maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46\" (UID: \"67332f71-74ce-4486-a341-cd32c7a3a9ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46" Oct 30 17:26:34.896856 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:34.896819 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67332f71-74ce-4486-a341-cd32c7a3a9ef-maas-api-tls\") pod \"maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46\" (UID: \"67332f71-74ce-4486-a341-cd32c7a3a9ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46" Oct 30 17:26:35.001054 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:35.000947 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67332f71-74ce-4486-a341-cd32c7a3a9ef-metrics-certs\") pod \"maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46\" (UID: \"67332f71-74ce-4486-a341-cd32c7a3a9ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46" Oct 30 17:26:35.001054 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:35.001017 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67332f71-74ce-4486-a341-cd32c7a3a9ef-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46\" (UID: \"67332f71-74ce-4486-a341-cd32c7a3a9ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46" Oct 30 17:26:35.001054 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:35.001055 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lblv\" (UniqueName: \"kubernetes.io/projected/67332f71-74ce-4486-a341-cd32c7a3a9ef-kube-api-access-2lblv\") pod \"maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46\" (UID: \"67332f71-74ce-4486-a341-cd32c7a3a9ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46" Oct 30 17:26:35.001368 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:35.001098 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67332f71-74ce-4486-a341-cd32c7a3a9ef-maas-api-tls\") pod \"maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46\" (UID: \"67332f71-74ce-4486-a341-cd32c7a3a9ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46" Oct 30 17:26:35.002361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:35.002331 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67332f71-74ce-4486-a341-cd32c7a3a9ef-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46\" (UID: \"67332f71-74ce-4486-a341-cd32c7a3a9ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46" Oct 30 17:26:35.003766 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:35.003745 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67332f71-74ce-4486-a341-cd32c7a3a9ef-maas-api-tls\") pod \"maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46\" (UID: \"67332f71-74ce-4486-a341-cd32c7a3a9ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46" Oct 30 17:26:35.004004 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:35.003982 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67332f71-74ce-4486-a341-cd32c7a3a9ef-metrics-certs\") pod \"maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46\" (UID: \"67332f71-74ce-4486-a341-cd32c7a3a9ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46" Oct 30 17:26:35.009648 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:35.009628 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w" Oct 30 17:26:35.025021 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:35.024969 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lblv\" (UniqueName: \"kubernetes.io/projected/67332f71-74ce-4486-a341-cd32c7a3a9ef-kube-api-access-2lblv\") pod \"maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46\" (UID: \"67332f71-74ce-4486-a341-cd32c7a3a9ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46" Oct 30 17:26:35.039605 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:35.039575 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm" event={"ID":"d9665073-6722-4274-bc2d-d453b06bd6c0","Type":"ContainerStarted","Data":"80d516712461197c10961cf3a0b8d86ff024c5cb3740af694dd0e7f8d213b0c0"} Oct 30 17:26:35.039740 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:35.039725 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm" Oct 30 17:26:35.101346 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:35.101308 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46" Oct 30 17:26:35.120293 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:35.120230 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm" podStartSLOduration=1.704133017 podStartE2EDuration="2.120210707s" totalImagesPullingTime="416.07769ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:35.119480072 +0000 UTC m=+853.171580005" watchObservedRunningTime="2026-10-30 17:26:35.120210707 +0000 UTC m=+853.172310642" Oct 30 17:26:35.436749 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:35.436719 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46"] Oct 30 17:26:35.438336 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:26:35.438310 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67332f71_74ce_4486_a341_cd32c7a3a9ef.slice/crio-81226f587be11aac89e4bc447e865eb537ba763da61e568ccd104c88e1ed1db8 WatchSource:0}: Error finding container 81226f587be11aac89e4bc447e865eb537ba763da61e568ccd104c88e1ed1db8: Status 404 returned error can't find the container with id 81226f587be11aac89e4bc447e865eb537ba763da61e568ccd104c88e1ed1db8 Oct 30 17:26:36.041567 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:26:36.041492 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:26:36.047846 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:36.047814 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46" event={"ID":"67332f71-74ce-4486-a341-cd32c7a3a9ef","Type":"ContainerStarted","Data":"31b8431181707a38db23e2dbb9e3ea33af7782f11f709dad2edb8503bd5ae6d8"} Oct 30 17:26:36.048009 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:36.047902 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46" event={"ID":"67332f71-74ce-4486-a341-cd32c7a3a9ef","Type":"ContainerStarted","Data":"81226f587be11aac89e4bc447e865eb537ba763da61e568ccd104c88e1ed1db8"} Oct 30 17:26:36.048009 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:36.047985 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46" Oct 30 17:26:36.110171 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:36.110097 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46" podStartSLOduration=1.7656381159999999 podStartE2EDuration="2.110076665s" totalImagesPullingTime="344.438549ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:36.108951332 +0000 UTC m=+854.161051291" watchObservedRunningTime="2026-10-30 17:26:36.110076665 +0000 UTC m=+854.162176599" Oct 30 17:26:38.955663 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:38.955612 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz"] Oct 30 17:26:38.960822 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:38.960792 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz" Oct 30 17:26:38.988987 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:38.988945 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz"] Oct 30 17:26:38.997107 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:38.997059 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-506124-serving-cert\"" Oct 30 17:26:39.078981 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:39.073695 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e25504b-8631-4770-9d29-a0ee77fd9116-maas-api-tls\") pod \"maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz\" (UID: \"3e25504b-8631-4770-9d29-a0ee77fd9116\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz" Oct 30 17:26:39.078981 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:39.074090 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e25504b-8631-4770-9d29-a0ee77fd9116-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz\" (UID: \"3e25504b-8631-4770-9d29-a0ee77fd9116\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz" Oct 30 17:26:39.078981 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:39.074216 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffz99\" (UniqueName: \"kubernetes.io/projected/3e25504b-8631-4770-9d29-a0ee77fd9116-kube-api-access-ffz99\") pod \"maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz\" (UID: \"3e25504b-8631-4770-9d29-a0ee77fd9116\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz" Oct 30 17:26:39.078981 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:39.074280 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e25504b-8631-4770-9d29-a0ee77fd9116-metrics-certs\") pod \"maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz\" (UID: \"3e25504b-8631-4770-9d29-a0ee77fd9116\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz" Oct 30 17:26:39.175570 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:39.175517 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e25504b-8631-4770-9d29-a0ee77fd9116-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz\" (UID: \"3e25504b-8631-4770-9d29-a0ee77fd9116\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz" Oct 30 17:26:39.175767 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:39.175616 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffz99\" (UniqueName: \"kubernetes.io/projected/3e25504b-8631-4770-9d29-a0ee77fd9116-kube-api-access-ffz99\") pod \"maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz\" (UID: \"3e25504b-8631-4770-9d29-a0ee77fd9116\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz" Oct 30 17:26:39.175767 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:39.175666 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e25504b-8631-4770-9d29-a0ee77fd9116-metrics-certs\") pod \"maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz\" (UID: \"3e25504b-8631-4770-9d29-a0ee77fd9116\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz" Oct 30 17:26:39.175767 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:39.175702 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e25504b-8631-4770-9d29-a0ee77fd9116-maas-api-tls\") pod \"maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz\" (UID: \"3e25504b-8631-4770-9d29-a0ee77fd9116\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz" Oct 30 17:26:39.177372 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:26:39.177123 4965 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-506124-serving-cert: secret "maas-api-e2e-worker-w2-506124-serving-cert" not found Oct 30 17:26:39.177372 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:26:39.177197 4965 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e25504b-8631-4770-9d29-a0ee77fd9116-maas-api-tls podName:3e25504b-8631-4770-9d29-a0ee77fd9116 nodeName:}" failed. No retries permitted until 2026-10-30 17:26:39.677177619 +0000 UTC m=+857.729277547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3e25504b-8631-4770-9d29-a0ee77fd9116-maas-api-tls") pod "maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz" (UID: "3e25504b-8631-4770-9d29-a0ee77fd9116") : secret "maas-api-e2e-worker-w2-506124-serving-cert" not found Oct 30 17:26:39.177372 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:39.177270 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e25504b-8631-4770-9d29-a0ee77fd9116-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz\" (UID: \"3e25504b-8631-4770-9d29-a0ee77fd9116\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz" Oct 30 17:26:39.179629 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:39.179582 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e25504b-8631-4770-9d29-a0ee77fd9116-metrics-certs\") pod \"maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz\" (UID: \"3e25504b-8631-4770-9d29-a0ee77fd9116\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz" Oct 30 17:26:39.199420 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:39.199357 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffz99\" (UniqueName: \"kubernetes.io/projected/3e25504b-8631-4770-9d29-a0ee77fd9116-kube-api-access-ffz99\") pod \"maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz\" (UID: \"3e25504b-8631-4770-9d29-a0ee77fd9116\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz" Oct 30 17:26:39.680780 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:39.680733 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e25504b-8631-4770-9d29-a0ee77fd9116-maas-api-tls\") pod \"maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz\" (UID: \"3e25504b-8631-4770-9d29-a0ee77fd9116\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz" Oct 30 17:26:39.683471 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:39.683440 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e25504b-8631-4770-9d29-a0ee77fd9116-maas-api-tls\") pod \"maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz\" (UID: \"3e25504b-8631-4770-9d29-a0ee77fd9116\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz" Oct 30 17:26:39.888320 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:39.888285 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz" Oct 30 17:26:40.020416 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:40.020366 4965 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:26:40.020944 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:40.020440 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" podUID="232650b5-3cab-4611-8ae6-9504a7ac2f04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:26:40.192949 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:40.192745 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz"] Oct 30 17:26:40.195846 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:26:40.195819 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e25504b_8631_4770_9d29_a0ee77fd9116.slice/crio-d50b4cb1986e4320f5b76254162611515338521857bc798091948aa2b3677cce WatchSource:0}: Error finding container d50b4cb1986e4320f5b76254162611515338521857bc798091948aa2b3677cce: Status 404 returned error can't find the container with id d50b4cb1986e4320f5b76254162611515338521857bc798091948aa2b3677cce Oct 30 17:26:41.053806 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:41.053718 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm" Oct 30 17:26:41.080031 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:41.079992 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz" event={"ID":"3e25504b-8631-4770-9d29-a0ee77fd9116","Type":"ContainerStarted","Data":"d5c7d0923ed7610a00d19bc664cfdd2f655aee9d05a66ee2a5e7ef5b476f3acb"} Oct 30 17:26:41.080169 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:41.080046 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz" event={"ID":"3e25504b-8631-4770-9d29-a0ee77fd9116","Type":"ContainerStarted","Data":"d50b4cb1986e4320f5b76254162611515338521857bc798091948aa2b3677cce"} Oct 30 17:26:41.080169 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:41.080092 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz" Oct 30 17:26:41.142899 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:41.140791 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz" podStartSLOduration=2.747723589 podStartE2EDuration="3.140771346s" totalImagesPullingTime="393.047757ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:41.14001032 +0000 UTC m=+859.192110255" watchObservedRunningTime="2026-10-30 17:26:41.140771346 +0000 UTC m=+859.192871276" Oct 30 17:26:41.264475 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:41.264439 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:26:42.022927 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:42.022900 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-27567a-7c6f9f9747-hhm7h" Oct 30 17:26:42.023153 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:42.022966 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-455086-6c947b47-fllqz" Oct 30 17:26:42.060701 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:42.060677 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46" Oct 30 17:26:42.088850 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:42.088806 4965 generic.go:356] "Generic (PLEG): container finished" podID="232650b5-3cab-4611-8ae6-9504a7ac2f04" containerID="df56035585263205e669e35fc96fa35e9dfd07c82318038fc3d0bc5eabc0cdac" exitCode=0 Oct 30 17:26:42.089056 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:42.088890 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" event={"ID":"232650b5-3cab-4611-8ae6-9504a7ac2f04","Type":"ContainerDied","Data":"df56035585263205e669e35fc96fa35e9dfd07c82318038fc3d0bc5eabc0cdac"} Oct 30 17:26:42.089444 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:42.089429 4965 scope.go:122] "RemoveContainer" containerID="df56035585263205e669e35fc96fa35e9dfd07c82318038fc3d0bc5eabc0cdac" Oct 30 17:26:42.090749 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:42.090719 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:26:42.090836 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:42.090786 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:26:42.090836 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:42.090819 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:26:43.096883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:43.096826 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" event={"ID":"232650b5-3cab-4611-8ae6-9504a7ac2f04","Type":"ContainerStarted","Data":"c65597114ab0a8e14f90974af8295469406297585fe8c86c2fbd41ac86d2fc0c"} Oct 30 17:26:43.378402 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:43.378263 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:43.378711 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:43.378689 4965 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 17:26:43.379148 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:43.379114 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" podUID="232650b5-3cab-4611-8ae6-9504a7ac2f04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 17:26:44.373887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:44.366204 4965 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 17:26:44.373887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:44.366263 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" podUID="232650b5-3cab-4611-8ae6-9504a7ac2f04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 17:26:45.047643 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:45.047613 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-506124-59bc6cd6c4-rp7pn" Oct 30 17:26:45.366887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:45.366159 4965 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 17:26:45.366887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:45.366239 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" podUID="232650b5-3cab-4611-8ae6-9504a7ac2f04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 17:26:46.373891 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:46.370174 4965 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 17:26:46.373891 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:46.370242 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" podUID="232650b5-3cab-4611-8ae6-9504a7ac2f04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 17:26:47.095062 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:47.095032 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz" Oct 30 17:26:47.175773 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:47.175735 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:26:47.367282 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:47.367196 4965 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 17:26:47.367528 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:47.367503 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" podUID="232650b5-3cab-4611-8ae6-9504a7ac2f04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 17:26:48.703908 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:48.703852 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:48.704373 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:48.704262 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:48.705144 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:48.705122 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:26:56.018063 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:56.018023 4965 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:26:56.018509 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:56.018109 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" podUID="ddb82e7c-5038-46b7-937e-c9ae4b12b2c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:26:58.178803 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:58.178773 4965 generic.go:356] "Generic (PLEG): container finished" podID="ddb82e7c-5038-46b7-937e-c9ae4b12b2c0" containerID="88b179d4d49aef2174d85adf10dae147bf57f623bdfb217c8831e1460c2fe2c2" exitCode=0 Oct 30 17:26:58.179266 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:58.178857 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" event={"ID":"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0","Type":"ContainerDied","Data":"88b179d4d49aef2174d85adf10dae147bf57f623bdfb217c8831e1460c2fe2c2"} Oct 30 17:26:58.179266 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:58.179262 4965 scope.go:122] "RemoveContainer" containerID="88b179d4d49aef2174d85adf10dae147bf57f623bdfb217c8831e1460c2fe2c2" Oct 30 17:26:58.180609 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:58.180583 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:26:58.180663 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:58.180650 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:26:58.180699 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:58.180685 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:26:59.192001 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:59.191966 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" event={"ID":"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0","Type":"ContainerStarted","Data":"7a6434102ff2f3ea0f60364af178fb96ef8e689579d38dcf8e72684777f76f3f"} Oct 30 17:26:59.483065 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:59.482961 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:26:59.924886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:59.924261 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:26:59.924886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:59.924389 4965 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 17:26:59.924886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:26:59.924433 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" podUID="ddb82e7c-5038-46b7-937e-c9ae4b12b2c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 17:27:00.919806 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:00.919771 4965 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 17:27:00.920213 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:00.919831 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" podUID="ddb82e7c-5038-46b7-937e-c9ae4b12b2c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 17:27:01.919772 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:01.919741 4965 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 17:27:01.919982 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:01.919792 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" podUID="ddb82e7c-5038-46b7-937e-c9ae4b12b2c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 17:27:02.920378 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:02.920348 4965 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 17:27:02.920907 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:02.920854 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" podUID="ddb82e7c-5038-46b7-937e-c9ae4b12b2c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 17:27:03.919443 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:03.919411 4965 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 17:27:03.919619 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:03.919462 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" podUID="ddb82e7c-5038-46b7-937e-c9ae4b12b2c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 17:27:04.923528 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:04.923497 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:27:04.924034 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:04.923773 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:27:04.924426 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:04.924411 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 17:27:06.008051 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:27:06.008018 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:27:11.230567 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:11.230525 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:27:14.330102 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:14.330071 4965 generic.go:356] "Generic (PLEG): container finished" podID="16d015c9-b3d7-45ae-a4f7-04aa68f678fd" containerID="3756862eda02255279216d61f3e436bd949ab882cb498a8403ebde16a3168476" exitCode=0 Oct 30 17:27:14.330576 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:14.330118 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" event={"ID":"16d015c9-b3d7-45ae-a4f7-04aa68f678fd","Type":"ContainerDied","Data":"3756862eda02255279216d61f3e436bd949ab882cb498a8403ebde16a3168476"} Oct 30 17:27:14.330576 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:14.330555 4965 scope.go:122] "RemoveContainer" containerID="3756862eda02255279216d61f3e436bd949ab882cb498a8403ebde16a3168476" Oct 30 17:27:14.331931 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:14.331900 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:27:14.332035 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:14.331988 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:27:14.332099 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:14.332038 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:27:15.337487 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:15.337449 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" event={"ID":"16d015c9-b3d7-45ae-a4f7-04aa68f678fd","Type":"ContainerStarted","Data":"ff301766bd8d874f52b0a792f0b64d7d642688e6dbf7b4acd1d777d72fdff97e"} Oct 30 17:27:15.639112 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:15.639039 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:27:15.639336 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:15.639311 4965 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 17:27:15.639386 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:15.639355 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" podUID="16d015c9-b3d7-45ae-a4f7-04aa68f678fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 17:27:16.639941 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:16.639907 4965 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 17:27:16.640343 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:16.639972 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" podUID="16d015c9-b3d7-45ae-a4f7-04aa68f678fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 17:27:17.639754 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:17.639714 4965 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 17:27:17.639995 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:17.639780 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" podUID="16d015c9-b3d7-45ae-a4f7-04aa68f678fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 17:27:18.640223 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:18.640190 4965 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 17:27:18.640634 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:18.640238 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" podUID="16d015c9-b3d7-45ae-a4f7-04aa68f678fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 17:27:19.639719 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:19.639683 4965 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 17:27:19.639921 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:19.639750 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" podUID="16d015c9-b3d7-45ae-a4f7-04aa68f678fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 17:27:20.686152 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:20.686113 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:27:20.686664 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:20.686536 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:27:20.687555 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:20.687531 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:27:20.986481 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:20.986402 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm"] Oct 30 17:27:20.990113 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:20.990091 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:27:21.007119 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:21.007094 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm"] Oct 30 17:27:21.008093 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:21.008069 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-8d946d\"/\"openshift-service-ca.crt\"" Oct 30 17:27:21.008337 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:21.008316 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-8d946d\"/\"kube-root-ca.crt\"" Oct 30 17:27:21.008475 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:21.008456 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-8d946d\"/\"default-dockercfg-hdh9p\"" Oct 30 17:27:21.008549 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:21.008528 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-8d946d\"/\"simulator-w6-8d946d-kserve-self-signed-certs\"" Oct 30 17:27:21.076599 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:21.076568 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f715fb84-3304-4d78-a4e6-f96c5b258e82-dshm\") pod \"simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm\" (UID: \"f715fb84-3304-4d78-a4e6-f96c5b258e82\") " pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:27:21.076599 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:21.076603 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv4rh\" (UniqueName: \"kubernetes.io/projected/f715fb84-3304-4d78-a4e6-f96c5b258e82-kube-api-access-kv4rh\") pod \"simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm\" (UID: \"f715fb84-3304-4d78-a4e6-f96c5b258e82\") " pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:27:21.076798 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:21.076761 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f715fb84-3304-4d78-a4e6-f96c5b258e82-model-cache\") pod \"simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm\" (UID: \"f715fb84-3304-4d78-a4e6-f96c5b258e82\") " pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:27:21.076798 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:21.076788 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f715fb84-3304-4d78-a4e6-f96c5b258e82-tls-certs\") pod \"simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm\" (UID: \"f715fb84-3304-4d78-a4e6-f96c5b258e82\") " pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:27:21.076921 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:21.076821 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f715fb84-3304-4d78-a4e6-f96c5b258e82-tmp-dir\") pod \"simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm\" (UID: \"f715fb84-3304-4d78-a4e6-f96c5b258e82\") " pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:27:21.076967 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:21.076935 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f715fb84-3304-4d78-a4e6-f96c5b258e82-home\") pod \"simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm\" (UID: \"f715fb84-3304-4d78-a4e6-f96c5b258e82\") " pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:27:21.177992 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:21.177963 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f715fb84-3304-4d78-a4e6-f96c5b258e82-tmp-dir\") pod \"simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm\" (UID: \"f715fb84-3304-4d78-a4e6-f96c5b258e82\") " pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:27:21.178187 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:21.178030 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f715fb84-3304-4d78-a4e6-f96c5b258e82-home\") pod \"simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm\" (UID: \"f715fb84-3304-4d78-a4e6-f96c5b258e82\") " pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:27:21.178187 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:21.178079 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f715fb84-3304-4d78-a4e6-f96c5b258e82-dshm\") pod \"simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm\" (UID: \"f715fb84-3304-4d78-a4e6-f96c5b258e82\") " pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:27:21.178187 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:21.178101 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv4rh\" (UniqueName: \"kubernetes.io/projected/f715fb84-3304-4d78-a4e6-f96c5b258e82-kube-api-access-kv4rh\") pod \"simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm\" (UID: \"f715fb84-3304-4d78-a4e6-f96c5b258e82\") " pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:27:21.178402 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:21.178276 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f715fb84-3304-4d78-a4e6-f96c5b258e82-model-cache\") pod \"simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm\" (UID: \"f715fb84-3304-4d78-a4e6-f96c5b258e82\") " pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:27:21.178402 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:21.178301 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f715fb84-3304-4d78-a4e6-f96c5b258e82-tls-certs\") pod \"simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm\" (UID: \"f715fb84-3304-4d78-a4e6-f96c5b258e82\") " pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:27:21.178668 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:21.178646 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f715fb84-3304-4d78-a4e6-f96c5b258e82-tmp-dir\") pod \"simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm\" (UID: \"f715fb84-3304-4d78-a4e6-f96c5b258e82\") " pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:27:21.178752 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:21.178680 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f715fb84-3304-4d78-a4e6-f96c5b258e82-home\") pod \"simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm\" (UID: \"f715fb84-3304-4d78-a4e6-f96c5b258e82\") " pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:27:21.178916 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:21.178890 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f715fb84-3304-4d78-a4e6-f96c5b258e82-model-cache\") pod \"simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm\" (UID: \"f715fb84-3304-4d78-a4e6-f96c5b258e82\") " pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:27:21.180668 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:21.180650 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f715fb84-3304-4d78-a4e6-f96c5b258e82-dshm\") pod \"simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm\" (UID: \"f715fb84-3304-4d78-a4e6-f96c5b258e82\") " pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:27:21.181244 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:21.181222 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f715fb84-3304-4d78-a4e6-f96c5b258e82-tls-certs\") pod \"simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm\" (UID: \"f715fb84-3304-4d78-a4e6-f96c5b258e82\") " pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:27:21.201445 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:21.201422 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv4rh\" (UniqueName: \"kubernetes.io/projected/f715fb84-3304-4d78-a4e6-f96c5b258e82-kube-api-access-kv4rh\") pod \"simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm\" (UID: \"f715fb84-3304-4d78-a4e6-f96c5b258e82\") " pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:27:21.314991 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:21.312077 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:27:21.622973 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:21.622943 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm"] Oct 30 17:27:21.624196 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:27:21.624170 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf715fb84_3304_4d78_a4e6_f96c5b258e82.slice/crio-895175ca4e7f6f4f2085df16437ae1d613240b35bb80f339db828bb6768b9391 WatchSource:0}: Error finding container 895175ca4e7f6f4f2085df16437ae1d613240b35bb80f339db828bb6768b9391: Status 404 returned error can't find the container with id 895175ca4e7f6f4f2085df16437ae1d613240b35bb80f339db828bb6768b9391 Oct 30 17:27:22.193174 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:22.193149 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 17:27:22.194728 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:22.194706 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 17:27:22.402897 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:22.402835 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" event={"ID":"f715fb84-3304-4d78-a4e6-f96c5b258e82","Type":"ContainerStarted","Data":"ae510ad22726505e73796748d152e754f92b6ff802f361288fdc07b7b9eaefb8"} Oct 30 17:27:22.402897 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:22.402912 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" event={"ID":"f715fb84-3304-4d78-a4e6-f96c5b258e82","Type":"ContainerStarted","Data":"895175ca4e7f6f4f2085df16437ae1d613240b35bb80f339db828bb6768b9391"} Oct 30 17:27:22.448085 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:22.447994 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podStartSLOduration=2.447981071 podStartE2EDuration="2.447981071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:22.447305584 +0000 UTC m=+900.499405539" watchObservedRunningTime="2026-10-30 17:27:22.447981071 +0000 UTC m=+900.500081054" Oct 30 17:27:23.312959 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:23.312924 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:27:24.221806 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:24.221775 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:27:31.313005 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:31.312972 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:27:31.325942 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:31.325915 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:27:31.460365 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:31.460333 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:27:35.996118 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:27:35.996079 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:27:36.211673 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:36.211641 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n"] Oct 30 17:27:36.214985 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:36.214967 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:27:36.244727 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:36.244685 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0e7403\"/\"kube-root-ca.crt\"" Oct 30 17:27:36.244727 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:36.244685 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0e7403\"/\"openshift-service-ca.crt\"" Oct 30 17:27:36.244727 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:36.244689 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0e7403\"/\"simulator-w4-0e7403-kserve-self-signed-certs\"" Oct 30 17:27:36.245015 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:36.244755 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0e7403\"/\"default-dockercfg-jgc4t\"" Oct 30 17:27:36.248701 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:36.248645 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n"] Oct 30 17:27:36.266991 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:36.266952 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13edf9da-7593-4b14-9830-f62cd466a6e4-tls-certs\") pod \"simulator-w4-0e7403-kserve-5668b8f548-5455n\" (UID: \"13edf9da-7593-4b14-9830-f62cd466a6e4\") " pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:27:36.267137 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:36.267014 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7kqs\" (UniqueName: \"kubernetes.io/projected/13edf9da-7593-4b14-9830-f62cd466a6e4-kube-api-access-z7kqs\") pod \"simulator-w4-0e7403-kserve-5668b8f548-5455n\" (UID: \"13edf9da-7593-4b14-9830-f62cd466a6e4\") " pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:27:36.267137 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:36.267103 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13edf9da-7593-4b14-9830-f62cd466a6e4-model-cache\") pod \"simulator-w4-0e7403-kserve-5668b8f548-5455n\" (UID: \"13edf9da-7593-4b14-9830-f62cd466a6e4\") " pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:27:36.267226 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:36.267211 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13edf9da-7593-4b14-9830-f62cd466a6e4-tmp-dir\") pod \"simulator-w4-0e7403-kserve-5668b8f548-5455n\" (UID: \"13edf9da-7593-4b14-9830-f62cd466a6e4\") " pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:27:36.267274 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:36.267260 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13edf9da-7593-4b14-9830-f62cd466a6e4-home\") pod \"simulator-w4-0e7403-kserve-5668b8f548-5455n\" (UID: \"13edf9da-7593-4b14-9830-f62cd466a6e4\") " pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:27:36.267310 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:36.267286 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13edf9da-7593-4b14-9830-f62cd466a6e4-dshm\") pod \"simulator-w4-0e7403-kserve-5668b8f548-5455n\" (UID: \"13edf9da-7593-4b14-9830-f62cd466a6e4\") " pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:27:36.369805 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:36.369772 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13edf9da-7593-4b14-9830-f62cd466a6e4-home\") pod \"simulator-w4-0e7403-kserve-5668b8f548-5455n\" (UID: \"13edf9da-7593-4b14-9830-f62cd466a6e4\") " pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:27:36.369994 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:36.369810 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13edf9da-7593-4b14-9830-f62cd466a6e4-dshm\") pod \"simulator-w4-0e7403-kserve-5668b8f548-5455n\" (UID: \"13edf9da-7593-4b14-9830-f62cd466a6e4\") " pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:27:36.369994 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:36.369902 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13edf9da-7593-4b14-9830-f62cd466a6e4-tls-certs\") pod \"simulator-w4-0e7403-kserve-5668b8f548-5455n\" (UID: \"13edf9da-7593-4b14-9830-f62cd466a6e4\") " pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:27:36.369994 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:36.369947 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7kqs\" (UniqueName: \"kubernetes.io/projected/13edf9da-7593-4b14-9830-f62cd466a6e4-kube-api-access-z7kqs\") pod \"simulator-w4-0e7403-kserve-5668b8f548-5455n\" (UID: \"13edf9da-7593-4b14-9830-f62cd466a6e4\") " pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:27:36.369994 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:36.369981 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13edf9da-7593-4b14-9830-f62cd466a6e4-model-cache\") pod \"simulator-w4-0e7403-kserve-5668b8f548-5455n\" (UID: \"13edf9da-7593-4b14-9830-f62cd466a6e4\") " pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:27:36.370199 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:36.370078 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13edf9da-7593-4b14-9830-f62cd466a6e4-tmp-dir\") pod \"simulator-w4-0e7403-kserve-5668b8f548-5455n\" (UID: \"13edf9da-7593-4b14-9830-f62cd466a6e4\") " pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:27:36.370273 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:36.370251 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13edf9da-7593-4b14-9830-f62cd466a6e4-home\") pod \"simulator-w4-0e7403-kserve-5668b8f548-5455n\" (UID: \"13edf9da-7593-4b14-9830-f62cd466a6e4\") " pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:27:36.370370 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:36.370334 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13edf9da-7593-4b14-9830-f62cd466a6e4-model-cache\") pod \"simulator-w4-0e7403-kserve-5668b8f548-5455n\" (UID: \"13edf9da-7593-4b14-9830-f62cd466a6e4\") " pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:27:36.370476 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:36.370380 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13edf9da-7593-4b14-9830-f62cd466a6e4-tmp-dir\") pod \"simulator-w4-0e7403-kserve-5668b8f548-5455n\" (UID: \"13edf9da-7593-4b14-9830-f62cd466a6e4\") " pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:27:36.372244 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:36.372221 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13edf9da-7593-4b14-9830-f62cd466a6e4-dshm\") pod \"simulator-w4-0e7403-kserve-5668b8f548-5455n\" (UID: \"13edf9da-7593-4b14-9830-f62cd466a6e4\") " pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:27:36.372397 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:36.372380 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13edf9da-7593-4b14-9830-f62cd466a6e4-tls-certs\") pod \"simulator-w4-0e7403-kserve-5668b8f548-5455n\" (UID: \"13edf9da-7593-4b14-9830-f62cd466a6e4\") " pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:27:36.393345 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:36.393311 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7kqs\" (UniqueName: \"kubernetes.io/projected/13edf9da-7593-4b14-9830-f62cd466a6e4-kube-api-access-z7kqs\") pod \"simulator-w4-0e7403-kserve-5668b8f548-5455n\" (UID: \"13edf9da-7593-4b14-9830-f62cd466a6e4\") " pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:27:36.526033 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:36.525953 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:27:36.825519 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:36.825488 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n"] Oct 30 17:27:36.827043 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:27:36.827017 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13edf9da_7593_4b14_9830_f62cd466a6e4.slice/crio-8ef70040331b11bdd90e60e6f59f8ffe589034d0b05f74a179e6b8e4c47f23a0 WatchSource:0}: Error finding container 8ef70040331b11bdd90e60e6f59f8ffe589034d0b05f74a179e6b8e4c47f23a0: Status 404 returned error can't find the container with id 8ef70040331b11bdd90e60e6f59f8ffe589034d0b05f74a179e6b8e4c47f23a0 Oct 30 17:27:37.488581 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:37.488543 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" event={"ID":"13edf9da-7593-4b14-9830-f62cd466a6e4","Type":"ContainerStarted","Data":"e5d52c9f64ca2e613bacee99ea76eaf6ddc3c839441e7fbfea139a464eb2e052"} Oct 30 17:27:37.488581 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:37.488589 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" event={"ID":"13edf9da-7593-4b14-9830-f62cd466a6e4","Type":"ContainerStarted","Data":"8ef70040331b11bdd90e60e6f59f8ffe589034d0b05f74a179e6b8e4c47f23a0"} Oct 30 17:27:37.527055 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:37.527022 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:27:37.533099 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:37.533055 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podStartSLOduration=1.5330407670000001 podStartE2EDuration="1.533040767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:37.531302185 +0000 UTC m=+915.583402116" watchObservedRunningTime="2026-10-30 17:27:37.533040767 +0000 UTC m=+915.585140698" Oct 30 17:27:40.227456 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:40.227421 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:27:46.526762 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:46.526727 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:27:46.539423 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:46.539393 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:27:46.551525 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:46.551505 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:27:50.786249 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:50.786216 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj"] Oct 30 17:27:50.790640 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:50.790617 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:27:50.808420 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:50.808389 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-455086\"/\"openshift-service-ca.crt\"" Oct 30 17:27:50.808564 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:50.808509 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-455086\"/\"kube-root-ca.crt\"" Oct 30 17:27:50.820403 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:50.820375 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-455086\"/\"premium-w1-455086-kserve-self-signed-certs\"" Oct 30 17:27:50.820556 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:50.820516 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-455086\"/\"default-dockercfg-dwx7b\"" Oct 30 17:27:50.821185 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:50.821164 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj"] Oct 30 17:27:50.938746 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:50.938709 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-dshm\") pod \"premium-w1-455086-kserve-747bfccf85-wv5dj\" (UID: \"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4\") " pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:27:50.938746 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:50.938746 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-model-cache\") pod \"premium-w1-455086-kserve-747bfccf85-wv5dj\" (UID: \"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4\") " pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:27:50.938999 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:50.938771 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-tmp-dir\") pod \"premium-w1-455086-kserve-747bfccf85-wv5dj\" (UID: \"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4\") " pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:27:50.938999 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:50.938962 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-tls-certs\") pod \"premium-w1-455086-kserve-747bfccf85-wv5dj\" (UID: \"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4\") " pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:27:50.939092 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:50.938999 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-home\") pod \"premium-w1-455086-kserve-747bfccf85-wv5dj\" (UID: \"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4\") " pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:27:50.939092 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:50.939028 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdsvx\" (UniqueName: \"kubernetes.io/projected/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-kube-api-access-vdsvx\") pod \"premium-w1-455086-kserve-747bfccf85-wv5dj\" (UID: \"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4\") " pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:27:51.040517 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:51.040433 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-dshm\") pod \"premium-w1-455086-kserve-747bfccf85-wv5dj\" (UID: \"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4\") " pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:27:51.040517 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:51.040470 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-model-cache\") pod \"premium-w1-455086-kserve-747bfccf85-wv5dj\" (UID: \"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4\") " pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:27:51.040517 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:51.040500 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-tmp-dir\") pod \"premium-w1-455086-kserve-747bfccf85-wv5dj\" (UID: \"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4\") " pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:27:51.040778 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:51.040551 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-tls-certs\") pod \"premium-w1-455086-kserve-747bfccf85-wv5dj\" (UID: \"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4\") " pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:27:51.040778 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:51.040585 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-home\") pod \"premium-w1-455086-kserve-747bfccf85-wv5dj\" (UID: \"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4\") " pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:27:51.040778 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:51.040737 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdsvx\" (UniqueName: \"kubernetes.io/projected/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-kube-api-access-vdsvx\") pod \"premium-w1-455086-kserve-747bfccf85-wv5dj\" (UID: \"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4\") " pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:27:51.040972 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:51.040949 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-model-cache\") pod \"premium-w1-455086-kserve-747bfccf85-wv5dj\" (UID: \"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4\") " pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:27:51.041032 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:51.040989 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-tmp-dir\") pod \"premium-w1-455086-kserve-747bfccf85-wv5dj\" (UID: \"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4\") " pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:27:51.041083 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:51.041038 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-home\") pod \"premium-w1-455086-kserve-747bfccf85-wv5dj\" (UID: \"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4\") " pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:27:51.042942 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:51.042921 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-dshm\") pod \"premium-w1-455086-kserve-747bfccf85-wv5dj\" (UID: \"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4\") " pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:27:51.043195 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:51.043177 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-tls-certs\") pod \"premium-w1-455086-kserve-747bfccf85-wv5dj\" (UID: \"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4\") " pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:27:51.063763 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:51.063741 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdsvx\" (UniqueName: \"kubernetes.io/projected/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-kube-api-access-vdsvx\") pod \"premium-w1-455086-kserve-747bfccf85-wv5dj\" (UID: \"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4\") " pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:27:51.106720 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:51.106696 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:27:51.412608 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:51.412581 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj"] Oct 30 17:27:51.413893 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:27:51.413847 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf9a4d4d_8cfa_42c8_a249_d2e1fa7deba4.slice/crio-e5a7923cb7f7a82602c342d86d1b98cb0409cb302d1dd09ff470aa81e4114f00 WatchSource:0}: Error finding container e5a7923cb7f7a82602c342d86d1b98cb0409cb302d1dd09ff470aa81e4114f00: Status 404 returned error can't find the container with id e5a7923cb7f7a82602c342d86d1b98cb0409cb302d1dd09ff470aa81e4114f00 Oct 30 17:27:51.570376 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:51.570343 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" event={"ID":"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4","Type":"ContainerStarted","Data":"51e78be71752c6b09881c3f34796c886b9a03d712a5ca0b5cde7e1d0d0a659c7"} Oct 30 17:27:51.570574 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:51.570394 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" event={"ID":"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4","Type":"ContainerStarted","Data":"e5a7923cb7f7a82602c342d86d1b98cb0409cb302d1dd09ff470aa81e4114f00"} Oct 30 17:27:51.614430 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:51.614379 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podStartSLOduration=1.6143655639999999 podStartE2EDuration="1.614365564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:51.613488002 +0000 UTC m=+929.665587951" watchObservedRunningTime="2026-10-30 17:27:51.614365564 +0000 UTC m=+929.666465494" Oct 30 17:27:52.107477 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:52.107443 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:27:55.118988 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:27:55.118953 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:28:01.107261 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:01.107224 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:28:01.120069 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:01.120038 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:28:01.638032 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:01.638004 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:28:06.007023 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:28:06.006927 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:28:07.514320 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:07.514283 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk"] Oct 30 17:28:07.518143 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:07.518122 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:07.547232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:07.547195 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22ed5344-88c7-4530-bcbf-327df41c4492-istio-podinfo\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:07.547390 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:07.547249 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-istio-data\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:07.547390 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:07.547353 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22ed5344-88c7-4530-bcbf-327df41c4492-istiod-ca-cert\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:07.547510 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:07.547442 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-istio-envoy\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:07.547510 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:07.547470 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb8td\" (UniqueName: \"kubernetes.io/projected/22ed5344-88c7-4530-bcbf-327df41c4492-kube-api-access-bb8td\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:07.547618 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:07.547532 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-workload-certs\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:07.547618 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:07.547584 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-credential-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:07.547727 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:07.547617 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22ed5344-88c7-4530-bcbf-327df41c4492-istio-token\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:07.547727 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:07.547654 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-workload-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:07.550385 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:07.550361 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-455086-openshift-default-dockercfg-98bfj\"" Oct 30 17:28:07.554553 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:07.554525 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk"] Oct 30 17:28:07.648409 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:07.648373 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22ed5344-88c7-4530-bcbf-327df41c4492-istio-podinfo\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:07.648409 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:07.648413 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-istio-data\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:07.648750 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:07.648467 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22ed5344-88c7-4530-bcbf-327df41c4492-istiod-ca-cert\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:07.648750 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:07.648553 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-istio-envoy\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:07.648750 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:07.648582 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb8td\" (UniqueName: \"kubernetes.io/projected/22ed5344-88c7-4530-bcbf-327df41c4492-kube-api-access-bb8td\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:07.648750 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:07.648636 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-workload-certs\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:07.648750 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:07.648687 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-credential-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:07.648750 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:07.648725 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22ed5344-88c7-4530-bcbf-327df41c4492-istio-token\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:07.649128 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:07.648757 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-workload-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:07.649128 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:07.648841 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-istio-data\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:07.649239 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:07.649135 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-workload-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:07.649239 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:07.649227 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-credential-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:07.649349 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:07.649244 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-workload-certs\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:07.649349 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:07.649340 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22ed5344-88c7-4530-bcbf-327df41c4492-istiod-ca-cert\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:07.650878 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:07.650845 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-istio-envoy\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:07.651008 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:07.650992 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22ed5344-88c7-4530-bcbf-327df41c4492-istio-podinfo\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:07.672220 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:07.672191 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22ed5344-88c7-4530-bcbf-327df41c4492-istio-token\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:07.672358 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:07.672224 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb8td\" (UniqueName: \"kubernetes.io/projected/22ed5344-88c7-4530-bcbf-327df41c4492-kube-api-access-bb8td\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:07.831946 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:07.831911 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:08.153093 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:08.153065 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk"] Oct 30 17:28:08.154975 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:28:08.154949 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22ed5344_88c7_4530_bcbf_327df41c4492.slice/crio-15ea0f01a19e4fe6adbf3b6c878d84c0859f704c5a7bcd276c3364e12bd7bb33 WatchSource:0}: Error finding container 15ea0f01a19e4fe6adbf3b6c878d84c0859f704c5a7bcd276c3364e12bd7bb33: Status 404 returned error can't find the container with id 15ea0f01a19e4fe6adbf3b6c878d84c0859f704c5a7bcd276c3364e12bd7bb33 Oct 30 17:28:08.157315 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:08.157279 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:28:08.157414 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:08.157366 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:28:08.157414 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:08.157398 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:28:08.686744 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:08.686705 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" event={"ID":"22ed5344-88c7-4530-bcbf-327df41c4492","Type":"ContainerStarted","Data":"e4b11dc2139671848fb8a7134b7ba9c9d287bc24862591f2c34cea0f32ea2cfc"} Oct 30 17:28:08.687373 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:08.687347 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" event={"ID":"22ed5344-88c7-4530-bcbf-327df41c4492","Type":"ContainerStarted","Data":"15ea0f01a19e4fe6adbf3b6c878d84c0859f704c5a7bcd276c3364e12bd7bb33"} Oct 30 17:28:08.777887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:08.775898 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" podStartSLOduration=1.77587856 podStartE2EDuration="1.77587856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:28:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:28:08.74618897 +0000 UTC m=+946.798288898" watchObservedRunningTime="2026-10-30 17:28:08.77587856 +0000 UTC m=+946.827978492" Oct 30 17:28:08.832208 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:08.832175 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:09.834608 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:09.834569 4965 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 17:28:09.834993 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:09.834636 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" podUID="22ed5344-88c7-4530-bcbf-327df41c4492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 17:28:10.833087 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:10.833055 4965 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 17:28:10.833261 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:10.833132 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" podUID="22ed5344-88c7-4530-bcbf-327df41c4492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 17:28:11.832907 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:11.832846 4965 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 17:28:11.833316 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:11.832939 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" podUID="22ed5344-88c7-4530-bcbf-327df41c4492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 17:28:12.832458 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:12.832424 4965 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 17:28:12.832640 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:12.832512 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" podUID="22ed5344-88c7-4530-bcbf-327df41c4492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 17:28:13.832956 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:13.832918 4965 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 17:28:13.833444 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:13.832978 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" podUID="22ed5344-88c7-4530-bcbf-327df41c4492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 17:28:14.972420 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:14.972380 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:14.972961 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:14.972901 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:14.974395 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:14.974372 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:28:15.435193 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:15.435158 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:28:22.313936 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.313904 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg"] Oct 30 17:28:22.318987 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.318970 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:22.350891 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.350836 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-27567a-openshift-default-dockercfg-cnckb\"" Oct 30 17:28:22.353637 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.353611 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg"] Oct 30 17:28:22.435626 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.435599 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-workload-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:22.435626 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.435629 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9157de31-170a-411a-9046-82c4d4f19231-istio-podinfo\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:22.435837 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.435653 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-istio-envoy\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:22.435837 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.435801 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9157de31-170a-411a-9046-82c4d4f19231-istio-token\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:22.435837 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.435835 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-istio-data\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:22.435978 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.435963 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-credential-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:22.436017 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.435989 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxtk5\" (UniqueName: \"kubernetes.io/projected/9157de31-170a-411a-9046-82c4d4f19231-kube-api-access-pxtk5\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:22.436055 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.436014 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-workload-certs\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:22.436055 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.436029 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9157de31-170a-411a-9046-82c4d4f19231-istiod-ca-cert\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:22.537314 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.537281 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-workload-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:22.537314 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.537321 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9157de31-170a-411a-9046-82c4d4f19231-istio-podinfo\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:22.537526 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.537349 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-istio-envoy\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:22.537526 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.537411 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9157de31-170a-411a-9046-82c4d4f19231-istio-token\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:22.537526 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.537433 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-istio-data\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:22.537656 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.537629 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-credential-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:22.537725 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.537682 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxtk5\" (UniqueName: \"kubernetes.io/projected/9157de31-170a-411a-9046-82c4d4f19231-kube-api-access-pxtk5\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:22.537790 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.537721 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-workload-certs\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:22.537790 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.537747 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9157de31-170a-411a-9046-82c4d4f19231-istiod-ca-cert\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:22.537944 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.537746 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-workload-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:22.538005 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.537979 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-istio-data\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:22.538158 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.538136 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-workload-certs\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:22.538229 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.538203 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-credential-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:22.538454 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.538432 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9157de31-170a-411a-9046-82c4d4f19231-istiod-ca-cert\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:22.539767 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.539746 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-istio-envoy\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:22.540053 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.540034 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9157de31-170a-411a-9046-82c4d4f19231-istio-podinfo\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:22.561378 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.561347 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9157de31-170a-411a-9046-82c4d4f19231-istio-token\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:22.561574 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.561555 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxtk5\" (UniqueName: \"kubernetes.io/projected/9157de31-170a-411a-9046-82c4d4f19231-kube-api-access-pxtk5\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:22.631881 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.631832 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:22.953513 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.953484 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg"] Oct 30 17:28:22.954690 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:28:22.954664 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9157de31_170a_411a_9046_82c4d4f19231.slice/crio-2c0da5d21a341f64ebfad65594a49edd6c10d0d9bf12cbdbb8704527bc603888 WatchSource:0}: Error finding container 2c0da5d21a341f64ebfad65594a49edd6c10d0d9bf12cbdbb8704527bc603888: Status 404 returned error can't find the container with id 2c0da5d21a341f64ebfad65594a49edd6c10d0d9bf12cbdbb8704527bc603888 Oct 30 17:28:22.957233 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.957197 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:28:22.957306 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.957265 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:28:22.957306 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:22.957297 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:28:23.787080 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:23.787039 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" event={"ID":"9157de31-170a-411a-9046-82c4d4f19231","Type":"ContainerStarted","Data":"a60c2596d9f7ba2f106503fcc14b1ebfa0b55eabd53cd362f829ca0a02e613ab"} Oct 30 17:28:23.787490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:23.787095 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" event={"ID":"9157de31-170a-411a-9046-82c4d4f19231","Type":"ContainerStarted","Data":"2c0da5d21a341f64ebfad65594a49edd6c10d0d9bf12cbdbb8704527bc603888"} Oct 30 17:28:23.882748 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:23.882669 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" podStartSLOduration=1.88264647 podStartE2EDuration="1.88264647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:28:23.855467133 +0000 UTC m=+961.907567096" watchObservedRunningTime="2026-10-30 17:28:23.88264647 +0000 UTC m=+961.934746402" Oct 30 17:28:24.632808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:24.632772 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:24.634358 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:24.634327 4965 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 17:28:24.634498 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:24.634375 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" podUID="9157de31-170a-411a-9046-82c4d4f19231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 17:28:25.633016 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:25.632983 4965 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 17:28:25.633408 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:25.633035 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" podUID="9157de31-170a-411a-9046-82c4d4f19231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 17:28:26.632649 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:26.632618 4965 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 17:28:26.632822 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:26.632681 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" podUID="9157de31-170a-411a-9046-82c4d4f19231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 17:28:27.632683 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:27.632648 4965 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 17:28:27.633117 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:27.632721 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" podUID="9157de31-170a-411a-9046-82c4d4f19231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 17:28:28.635319 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:28.634551 4965 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 17:28:28.635319 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:28.634612 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" podUID="9157de31-170a-411a-9046-82c4d4f19231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 17:28:29.637533 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:29.637501 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:29.638069 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:29.637892 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:29.639361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:29.639333 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:28:35.996736 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:28:35.996683 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:28:42.059447 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:42.059410 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:28:52.915446 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:52.915409 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll"] Oct 30 17:28:52.921313 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:52.921289 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:28:52.940439 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:52.940409 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-306aed-openshift-default-dockercfg-rbbs2\"" Oct 30 17:28:52.998398 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:52.998365 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll"] Oct 30 17:28:53.038401 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:53.038359 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istiod-ca-cert\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:28:53.038784 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:53.038761 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istio-envoy\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:28:53.039021 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:53.039003 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istio-token\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:28:53.039228 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:53.039211 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-workload-certs\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:28:53.039425 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:53.039408 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istio-podinfo\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:28:53.039591 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:53.039574 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7g7x\" (UniqueName: \"kubernetes.io/projected/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-kube-api-access-k7g7x\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:28:53.039773 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:53.039756 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-workload-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:28:53.039976 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:53.039957 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-credential-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:28:53.040148 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:53.040131 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istio-data\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:28:53.150640 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:53.150599 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istio-envoy\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:28:53.155024 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:53.154991 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istio-token\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:28:53.155293 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:53.155265 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-workload-certs\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:28:53.155438 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:53.155425 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istio-podinfo\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:28:53.155556 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:53.155543 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7g7x\" (UniqueName: \"kubernetes.io/projected/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-kube-api-access-k7g7x\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:28:53.155669 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:53.155657 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-workload-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:28:53.163513 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:53.155804 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-credential-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:28:53.171988 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:53.163713 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istio-data\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:28:53.172375 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:53.172266 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istiod-ca-cert\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:28:53.172918 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:53.172900 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-credential-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:28:53.173626 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:53.173606 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istio-data\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:28:53.181626 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:53.181607 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-workload-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:28:53.191902 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:53.182370 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-workload-certs\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:28:53.192927 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:53.192908 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istiod-ca-cert\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:28:53.216600 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:53.216575 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istio-podinfo\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:28:53.227793 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:53.227730 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7g7x\" (UniqueName: \"kubernetes.io/projected/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-kube-api-access-k7g7x\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:28:53.253069 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:53.253022 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istio-envoy\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:28:53.260903 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:53.255266 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istio-token\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:28:53.542395 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:53.542084 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:28:54.117420 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:54.117390 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll"] Oct 30 17:28:54.121688 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:54.121647 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:28:54.121786 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:54.121738 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:28:54.121832 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:54.121784 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:28:55.009887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:55.000937 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" event={"ID":"f170ecd7-a0bf-469a-b3ac-cbd739327d4d","Type":"ContainerStarted","Data":"4a6d4e5b44446802b627b278445a8fb338e3d490ce429967110336398727914b"} Oct 30 17:28:55.009887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:55.001009 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" event={"ID":"f170ecd7-a0bf-469a-b3ac-cbd739327d4d","Type":"ContainerStarted","Data":"414f5abbfd735e3d3cac69dd4e8533143c1d2286b206f1504e0f1ae3993458a3"} Oct 30 17:28:55.104886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:55.103384 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" podStartSLOduration=3.103360606 podStartE2EDuration="3.103360606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:28:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:28:55.068490605 +0000 UTC m=+993.120590537" watchObservedRunningTime="2026-10-30 17:28:55.103360606 +0000 UTC m=+993.155460538" Oct 30 17:28:55.544991 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:55.543096 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:28:55.544991 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:55.544410 4965 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 17:28:55.544991 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:55.544456 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" podUID="f170ecd7-a0bf-469a-b3ac-cbd739327d4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 17:28:56.543332 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:56.543293 4965 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 17:28:56.543510 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:56.543354 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" podUID="f170ecd7-a0bf-469a-b3ac-cbd739327d4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 17:28:57.544012 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:57.543978 4965 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 17:28:57.544696 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:57.544666 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" podUID="f170ecd7-a0bf-469a-b3ac-cbd739327d4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 17:28:58.543011 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:58.542971 4965 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 17:28:58.543192 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:58.543026 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" podUID="f170ecd7-a0bf-469a-b3ac-cbd739327d4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 17:28:59.543473 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:59.543441 4965 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 17:28:59.543900 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:28:59.543492 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" podUID="f170ecd7-a0bf-469a-b3ac-cbd739327d4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 17:29:00.542972 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:00.542942 4965 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 17:29:00.543149 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:00.542990 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" podUID="f170ecd7-a0bf-469a-b3ac-cbd739327d4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 17:29:01.547919 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:01.547853 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:29:01.548360 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:01.548343 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:29:01.549017 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:01.548994 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 17:29:05.729254 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:05.729213 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:29:05.996037 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:29:05.995960 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:29:12.642498 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.642465 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774"] Oct 30 17:29:12.646486 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.646463 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:12.674691 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.674666 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5"] Oct 30 17:29:12.678707 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.678681 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:12.692369 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.692339 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774"] Oct 30 17:29:12.711518 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.711482 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5"] Oct 30 17:29:12.760369 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.760340 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-rh774\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:12.760532 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.760375 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-rh774\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:12.760532 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.760491 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-rh774\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:12.760532 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.760522 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dg4j\" (UniqueName: \"kubernetes.io/projected/337938ce-7eb2-45bf-a5aa-0ef36cee9444-kube-api-access-7dg4j\") pod \"maas-default-gateway-openshift-default-7df854bf67-rh774\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:12.760651 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.760562 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-rh774\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:12.760717 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.760694 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-rh774\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:12.760753 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.760728 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-rh774\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:12.760788 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.760761 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-rh774\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:12.760830 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.760818 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-rh774\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:12.861957 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.861925 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a692092-9a09-4a93-b285-675f8268eb68-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-zv8m5\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:12.861957 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.861959 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnq64\" (UniqueName: \"kubernetes.io/projected/0a692092-9a09-4a93-b285-675f8268eb68-kube-api-access-tnq64\") pod \"maas-default-gateway-openshift-default-7df854bf67-zv8m5\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:12.862148 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.861985 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-rh774\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:12.862148 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.862100 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-rh774\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:12.862232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.862144 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a692092-9a09-4a93-b285-675f8268eb68-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-zv8m5\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:12.862232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.862191 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-zv8m5\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:12.862296 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.862237 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-zv8m5\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:12.862330 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.862297 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-zv8m5\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:12.862415 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.862397 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-rh774\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:12.862452 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.862430 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-zv8m5\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:12.862494 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.862457 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-rh774\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:12.862494 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.862464 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dg4j\" (UniqueName: \"kubernetes.io/projected/337938ce-7eb2-45bf-a5aa-0ef36cee9444-kube-api-access-7dg4j\") pod \"maas-default-gateway-openshift-default-7df854bf67-rh774\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:12.862644 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.862629 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-rh774\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:12.862644 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.862625 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a692092-9a09-4a93-b285-675f8268eb68-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-zv8m5\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:12.862743 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.862697 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-rh774\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:12.862778 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.862749 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-rh774\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:12.862816 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.862776 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-rh774\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:12.862888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.862819 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-rh774\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:12.862888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.862852 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-zv8m5\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:12.863000 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.862939 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-rh774\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:12.863168 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.863135 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-rh774\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:12.863340 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.863318 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-rh774\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:12.863413 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.863390 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-rh774\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:12.864511 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.864486 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-rh774\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:12.865087 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.865072 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-rh774\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:12.886737 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.886712 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-rh774\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:12.887108 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.887088 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dg4j\" (UniqueName: \"kubernetes.io/projected/337938ce-7eb2-45bf-a5aa-0ef36cee9444-kube-api-access-7dg4j\") pod \"maas-default-gateway-openshift-default-7df854bf67-rh774\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:12.960539 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.960464 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:12.964039 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.964014 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a692092-9a09-4a93-b285-675f8268eb68-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-zv8m5\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:12.964123 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.964053 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnq64\" (UniqueName: \"kubernetes.io/projected/0a692092-9a09-4a93-b285-675f8268eb68-kube-api-access-tnq64\") pod \"maas-default-gateway-openshift-default-7df854bf67-zv8m5\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:12.964123 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.964091 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a692092-9a09-4a93-b285-675f8268eb68-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-zv8m5\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:12.964212 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.964120 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-zv8m5\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:12.964212 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.964150 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-zv8m5\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:12.964212 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.964196 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-zv8m5\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:12.964358 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.964255 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-zv8m5\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:12.964358 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.964300 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a692092-9a09-4a93-b285-675f8268eb68-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-zv8m5\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:12.964474 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.964370 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-zv8m5\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:12.964720 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.964628 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-zv8m5\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:12.965283 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.964843 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-zv8m5\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:12.965283 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.964885 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-zv8m5\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:12.965283 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.965022 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-zv8m5\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:12.965283 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.965055 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a692092-9a09-4a93-b285-675f8268eb68-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-zv8m5\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:12.967018 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.966990 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-zv8m5\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:12.967150 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.967132 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a692092-9a09-4a93-b285-675f8268eb68-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-zv8m5\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:12.988269 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.988239 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a692092-9a09-4a93-b285-675f8268eb68-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-zv8m5\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:12.988478 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.988458 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnq64\" (UniqueName: \"kubernetes.io/projected/0a692092-9a09-4a93-b285-675f8268eb68-kube-api-access-tnq64\") pod \"maas-default-gateway-openshift-default-7df854bf67-zv8m5\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:12.995277 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:12.995257 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:13.283335 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:13.283305 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774"] Oct 30 17:29:13.284541 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:29:13.284517 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod337938ce_7eb2_45bf_a5aa_0ef36cee9444.slice/crio-ec369bb42765200f9654e0bd2cc0db0e3dcdf0c8d4b94dc042f37f8a2f60faf4 WatchSource:0}: Error finding container ec369bb42765200f9654e0bd2cc0db0e3dcdf0c8d4b94dc042f37f8a2f60faf4: Status 404 returned error can't find the container with id ec369bb42765200f9654e0bd2cc0db0e3dcdf0c8d4b94dc042f37f8a2f60faf4 Oct 30 17:29:13.286853 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:13.286817 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:29:13.286937 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:13.286904 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:29:13.286937 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:13.286932 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:29:13.338996 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:13.326784 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5"] Oct 30 17:29:13.338996 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:13.338176 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:29:13.338996 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:13.338313 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:29:13.338996 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:13.338393 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:29:14.186885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:14.185185 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" event={"ID":"337938ce-7eb2-45bf-a5aa-0ef36cee9444","Type":"ContainerStarted","Data":"88bb0c7d08d4ec1e0f0add33fa0bfda8420f4953b1d3ad353f0f1fa163594a97"} Oct 30 17:29:14.186885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:14.185251 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" event={"ID":"337938ce-7eb2-45bf-a5aa-0ef36cee9444","Type":"ContainerStarted","Data":"ec369bb42765200f9654e0bd2cc0db0e3dcdf0c8d4b94dc042f37f8a2f60faf4"} Oct 30 17:29:14.206068 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:14.205201 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" event={"ID":"0a692092-9a09-4a93-b285-675f8268eb68","Type":"ContainerStarted","Data":"8047491db76a2f38b2c4657a8a823a1aaadd26f2ba35cbe6992ddf36d82d1e91"} Oct 30 17:29:14.206068 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:14.205272 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" event={"ID":"0a692092-9a09-4a93-b285-675f8268eb68","Type":"ContainerStarted","Data":"39e8282a6e83d06085e0f8612cd129e1fe467156fd08e1569e5e8436397af189"} Oct 30 17:29:14.247898 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:14.247835 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" podStartSLOduration=2.247821272 podStartE2EDuration="2.247821272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:14.24689566 +0000 UTC m=+1012.298995590" watchObservedRunningTime="2026-10-30 17:29:14.247821272 +0000 UTC m=+1012.299921249" Oct 30 17:29:14.327908 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:14.325531 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" podStartSLOduration=2.325516422 podStartE2EDuration="2.325516422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:14.324193617 +0000 UTC m=+1012.376293547" watchObservedRunningTime="2026-10-30 17:29:14.325516422 +0000 UTC m=+1012.377616352" Oct 30 17:29:14.961091 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:14.961040 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:14.965886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:14.963463 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rh774 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 17:29:14.965886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:14.963531 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" podUID="337938ce-7eb2-45bf-a5aa-0ef36cee9444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 17:29:15.006890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:14.997119 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:15.006890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:14.997234 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-zv8m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 17:29:15.006890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:14.997274 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" podUID="0a692092-9a09-4a93-b285-675f8268eb68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 17:29:15.983887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:15.983313 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rh774 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 17:29:15.983887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:15.983374 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" podUID="337938ce-7eb2-45bf-a5aa-0ef36cee9444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 17:29:16.006885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:16.006815 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-zv8m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 17:29:16.007094 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:16.006898 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" podUID="0a692092-9a09-4a93-b285-675f8268eb68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 17:29:16.964306 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:16.964267 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rh774 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 17:29:16.964585 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:16.964555 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" podUID="337938ce-7eb2-45bf-a5aa-0ef36cee9444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 17:29:16.999266 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:16.999230 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-zv8m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 17:29:16.999853 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:16.999815 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" podUID="0a692092-9a09-4a93-b285-675f8268eb68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 17:29:17.961434 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:17.961394 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rh774 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 17:29:17.961750 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:17.961716 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" podUID="337938ce-7eb2-45bf-a5aa-0ef36cee9444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 17:29:18.003763 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:18.003728 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-zv8m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 17:29:18.004425 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:18.004377 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" podUID="0a692092-9a09-4a93-b285-675f8268eb68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 17:29:18.961315 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:18.961275 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rh774 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 17:29:18.961647 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:18.961606 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" podUID="337938ce-7eb2-45bf-a5aa-0ef36cee9444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 17:29:19.002345 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:19.002306 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-zv8m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 17:29:19.002630 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:19.002601 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" podUID="0a692092-9a09-4a93-b285-675f8268eb68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 17:29:19.961243 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:19.961206 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rh774 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 17:29:19.961658 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:19.961269 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" podUID="337938ce-7eb2-45bf-a5aa-0ef36cee9444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 17:29:19.995857 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:19.995820 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-zv8m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 17:29:19.996036 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:19.995906 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" podUID="0a692092-9a09-4a93-b285-675f8268eb68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 17:29:20.961490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:20.961452 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rh774 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 17:29:20.961895 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:20.961519 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" podUID="337938ce-7eb2-45bf-a5aa-0ef36cee9444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 17:29:20.995696 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:20.995661 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-zv8m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 17:29:20.995854 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:20.995710 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" podUID="0a692092-9a09-4a93-b285-675f8268eb68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 17:29:21.973847 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:21.973813 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:21.975106 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:21.975084 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:21.976103 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:21.976077 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:29:21.999603 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:21.999582 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:21.999900 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:21.999856 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:22.000529 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:22.000512 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:29:35.997310 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:29:35.997232 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:29:36.316413 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:36.316328 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:29:37.810549 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:37.810488 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx"] Oct 30 17:29:37.817608 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:37.817522 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:37.834739 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:37.834714 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx"] Oct 30 17:29:37.929513 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:37.929477 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-workload-certs\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:37.929708 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:37.929551 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4d48ca-a065-4a13-89a7-efcc2301046f-istiod-ca-cert\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:37.929708 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:37.929648 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh9nl\" (UniqueName: \"kubernetes.io/projected/3f4d48ca-a065-4a13-89a7-efcc2301046f-kube-api-access-dh9nl\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:37.929838 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:37.929741 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-workload-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:37.929838 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:37.929791 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-istio-data\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:37.929965 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:37.929880 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-istio-envoy\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:37.929965 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:37.929935 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-credential-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:37.930040 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:37.930015 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f4d48ca-a065-4a13-89a7-efcc2301046f-istio-token\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:37.930076 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:37.930050 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4d48ca-a065-4a13-89a7-efcc2301046f-istio-podinfo\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:38.030953 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.030918 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh9nl\" (UniqueName: \"kubernetes.io/projected/3f4d48ca-a065-4a13-89a7-efcc2301046f-kube-api-access-dh9nl\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:38.031147 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.031003 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-workload-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:38.031147 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.031029 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-istio-data\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:38.031147 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.031051 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-istio-envoy\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:38.031147 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.031102 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-credential-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:38.031361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.031168 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f4d48ca-a065-4a13-89a7-efcc2301046f-istio-token\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:38.031361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.031193 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4d48ca-a065-4a13-89a7-efcc2301046f-istio-podinfo\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:38.031361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.031249 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-workload-certs\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:38.031361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.031300 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4d48ca-a065-4a13-89a7-efcc2301046f-istiod-ca-cert\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:38.031546 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.031479 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-workload-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:38.031546 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.031534 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-istio-data\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:38.031639 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.031616 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-credential-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:38.031831 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.031807 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-workload-certs\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:38.031989 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.031936 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4d48ca-a065-4a13-89a7-efcc2301046f-istiod-ca-cert\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:38.033553 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.033532 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-istio-envoy\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:38.033688 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.033670 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4d48ca-a065-4a13-89a7-efcc2301046f-istio-podinfo\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:38.054251 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.054219 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f4d48ca-a065-4a13-89a7-efcc2301046f-istio-token\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:38.054364 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.054315 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh9nl\" (UniqueName: \"kubernetes.io/projected/3f4d48ca-a065-4a13-89a7-efcc2301046f-kube-api-access-dh9nl\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:38.136029 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.136001 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:38.456204 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.456162 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx"] Oct 30 17:29:38.459166 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:29:38.459140 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f4d48ca_a065_4a13_89a7_efcc2301046f.slice/crio-84cd380b4683cef8dc9acf1a2589ccb1a1e24a341ebab21dcf48702c8ce0b8ac WatchSource:0}: Error finding container 84cd380b4683cef8dc9acf1a2589ccb1a1e24a341ebab21dcf48702c8ce0b8ac: Status 404 returned error can't find the container with id 84cd380b4683cef8dc9acf1a2589ccb1a1e24a341ebab21dcf48702c8ce0b8ac Oct 30 17:29:38.462357 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.462321 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:29:38.462450 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.462388 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:29:38.462450 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.462416 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:29:38.826547 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.826521 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll"] Oct 30 17:29:38.830527 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.830508 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:29:38.849933 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.849901 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-db98b0\"/\"kube-root-ca.crt\"" Oct 30 17:29:38.862188 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.862143 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-db98b0\"/\"openshift-service-ca.crt\"" Oct 30 17:29:38.862188 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.862178 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-db98b0\"/\"distinct-w5-db98b0-kserve-self-signed-certs\"" Oct 30 17:29:38.862437 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.862143 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-db98b0\"/\"default-dockercfg-hnr9h\"" Oct 30 17:29:38.865655 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.865625 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll"] Oct 30 17:29:38.943530 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.943494 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46254fda-8014-4da3-9b02-beb7a1a1418c-model-cache\") pod \"distinct-w5-db98b0-kserve-7d59d9bddf-ljzll\" (UID: \"46254fda-8014-4da3-9b02-beb7a1a1418c\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:29:38.943736 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.943712 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkx7k\" (UniqueName: \"kubernetes.io/projected/46254fda-8014-4da3-9b02-beb7a1a1418c-kube-api-access-dkx7k\") pod \"distinct-w5-db98b0-kserve-7d59d9bddf-ljzll\" (UID: \"46254fda-8014-4da3-9b02-beb7a1a1418c\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:29:38.943801 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.943764 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46254fda-8014-4da3-9b02-beb7a1a1418c-home\") pod \"distinct-w5-db98b0-kserve-7d59d9bddf-ljzll\" (UID: \"46254fda-8014-4da3-9b02-beb7a1a1418c\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:29:38.943902 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.943833 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46254fda-8014-4da3-9b02-beb7a1a1418c-tls-certs\") pod \"distinct-w5-db98b0-kserve-7d59d9bddf-ljzll\" (UID: \"46254fda-8014-4da3-9b02-beb7a1a1418c\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:29:38.943902 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.943857 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46254fda-8014-4da3-9b02-beb7a1a1418c-dshm\") pod \"distinct-w5-db98b0-kserve-7d59d9bddf-ljzll\" (UID: \"46254fda-8014-4da3-9b02-beb7a1a1418c\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:29:38.943984 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:38.943926 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46254fda-8014-4da3-9b02-beb7a1a1418c-tmp-dir\") pod \"distinct-w5-db98b0-kserve-7d59d9bddf-ljzll\" (UID: \"46254fda-8014-4da3-9b02-beb7a1a1418c\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:29:39.048814 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:39.047935 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkx7k\" (UniqueName: \"kubernetes.io/projected/46254fda-8014-4da3-9b02-beb7a1a1418c-kube-api-access-dkx7k\") pod \"distinct-w5-db98b0-kserve-7d59d9bddf-ljzll\" (UID: \"46254fda-8014-4da3-9b02-beb7a1a1418c\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:29:39.048814 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:39.047992 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46254fda-8014-4da3-9b02-beb7a1a1418c-home\") pod \"distinct-w5-db98b0-kserve-7d59d9bddf-ljzll\" (UID: \"46254fda-8014-4da3-9b02-beb7a1a1418c\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:29:39.048814 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:39.048023 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46254fda-8014-4da3-9b02-beb7a1a1418c-tls-certs\") pod \"distinct-w5-db98b0-kserve-7d59d9bddf-ljzll\" (UID: \"46254fda-8014-4da3-9b02-beb7a1a1418c\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:29:39.048814 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:39.048047 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46254fda-8014-4da3-9b02-beb7a1a1418c-dshm\") pod \"distinct-w5-db98b0-kserve-7d59d9bddf-ljzll\" (UID: \"46254fda-8014-4da3-9b02-beb7a1a1418c\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:29:39.048814 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:39.048069 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46254fda-8014-4da3-9b02-beb7a1a1418c-tmp-dir\") pod \"distinct-w5-db98b0-kserve-7d59d9bddf-ljzll\" (UID: \"46254fda-8014-4da3-9b02-beb7a1a1418c\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:29:39.048814 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:39.048168 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46254fda-8014-4da3-9b02-beb7a1a1418c-model-cache\") pod \"distinct-w5-db98b0-kserve-7d59d9bddf-ljzll\" (UID: \"46254fda-8014-4da3-9b02-beb7a1a1418c\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:29:39.048814 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:39.048729 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46254fda-8014-4da3-9b02-beb7a1a1418c-model-cache\") pod \"distinct-w5-db98b0-kserve-7d59d9bddf-ljzll\" (UID: \"46254fda-8014-4da3-9b02-beb7a1a1418c\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:29:39.052899 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:39.049353 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46254fda-8014-4da3-9b02-beb7a1a1418c-home\") pod \"distinct-w5-db98b0-kserve-7d59d9bddf-ljzll\" (UID: \"46254fda-8014-4da3-9b02-beb7a1a1418c\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:29:39.055896 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:39.055853 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46254fda-8014-4da3-9b02-beb7a1a1418c-tls-certs\") pod \"distinct-w5-db98b0-kserve-7d59d9bddf-ljzll\" (UID: \"46254fda-8014-4da3-9b02-beb7a1a1418c\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:29:39.077665 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:39.077594 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46254fda-8014-4da3-9b02-beb7a1a1418c-tmp-dir\") pod \"distinct-w5-db98b0-kserve-7d59d9bddf-ljzll\" (UID: \"46254fda-8014-4da3-9b02-beb7a1a1418c\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:29:39.129719 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:39.129684 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46254fda-8014-4da3-9b02-beb7a1a1418c-dshm\") pod \"distinct-w5-db98b0-kserve-7d59d9bddf-ljzll\" (UID: \"46254fda-8014-4da3-9b02-beb7a1a1418c\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:29:39.141655 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:39.141623 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkx7k\" (UniqueName: \"kubernetes.io/projected/46254fda-8014-4da3-9b02-beb7a1a1418c-kube-api-access-dkx7k\") pod \"distinct-w5-db98b0-kserve-7d59d9bddf-ljzll\" (UID: \"46254fda-8014-4da3-9b02-beb7a1a1418c\") " pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:29:39.152516 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:39.152483 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:29:39.412307 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:39.412217 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" event={"ID":"3f4d48ca-a065-4a13-89a7-efcc2301046f","Type":"ContainerStarted","Data":"31f5b061eb9dd73509ab603cb30a25db20fc90ad4a8f6bff85f2e069b2332034"} Oct 30 17:29:39.412307 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:39.412304 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" event={"ID":"3f4d48ca-a065-4a13-89a7-efcc2301046f","Type":"ContainerStarted","Data":"84cd380b4683cef8dc9acf1a2589ccb1a1e24a341ebab21dcf48702c8ce0b8ac"} Oct 30 17:29:39.487369 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:39.487309 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" podStartSLOduration=2.487288478 podStartE2EDuration="2.487288478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:39.484121147 +0000 UTC m=+1037.536221098" watchObservedRunningTime="2026-10-30 17:29:39.487288478 +0000 UTC m=+1037.539388411" Oct 30 17:29:39.489389 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:29:39.489357 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46254fda_8014_4da3_9b02_beb7a1a1418c.slice/crio-13a90a7d8c93bc6e6404f42f03ccb9288a1243208d13f3f4901b57657da9c158 WatchSource:0}: Error finding container 13a90a7d8c93bc6e6404f42f03ccb9288a1243208d13f3f4901b57657da9c158: Status 404 returned error can't find the container with id 13a90a7d8c93bc6e6404f42f03ccb9288a1243208d13f3f4901b57657da9c158 Oct 30 17:29:39.490961 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:39.490942 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll"] Oct 30 17:29:40.136544 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:40.136511 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:40.137976 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:40.137952 4965 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 17:29:40.138092 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:40.138004 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" podUID="3f4d48ca-a065-4a13-89a7-efcc2301046f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 17:29:40.419585 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:40.419498 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" event={"ID":"46254fda-8014-4da3-9b02-beb7a1a1418c","Type":"ContainerStarted","Data":"22c05af1832435c89516450b0cbc3a88eff16b9877d5c2cfc707f8703c65de95"} Oct 30 17:29:40.419585 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:40.419542 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" event={"ID":"46254fda-8014-4da3-9b02-beb7a1a1418c","Type":"ContainerStarted","Data":"13a90a7d8c93bc6e6404f42f03ccb9288a1243208d13f3f4901b57657da9c158"} Oct 30 17:29:40.481923 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:40.463224 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podStartSLOduration=2.4632048 podStartE2EDuration="2.4632048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:40.461214709 +0000 UTC m=+1038.513314668" watchObservedRunningTime="2026-10-30 17:29:40.4632048 +0000 UTC m=+1038.515304732" Oct 30 17:29:41.136640 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:41.136602 4965 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 17:29:41.137047 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:41.136653 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" podUID="3f4d48ca-a065-4a13-89a7-efcc2301046f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 17:29:41.152779 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:41.152754 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:29:42.137186 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:42.137153 4965 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 17:29:42.137596 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:42.137202 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" podUID="3f4d48ca-a065-4a13-89a7-efcc2301046f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 17:29:43.136889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:43.136805 4965 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 17:29:43.137098 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:43.136897 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" podUID="3f4d48ca-a065-4a13-89a7-efcc2301046f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 17:29:43.810503 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:43.810463 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7"] Oct 30 17:29:43.815313 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:43.815290 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:29:43.834677 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:43.834648 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7"] Oct 30 17:29:43.842693 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:43.842642 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-27567a\"/\"openshift-service-ca.crt\"" Oct 30 17:29:43.842693 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:43.842650 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-27567a\"/\"kube-root-ca.crt\"" Oct 30 17:29:43.842927 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:43.842735 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-27567a\"/\"distinct-w0-27567a-kserve-self-signed-certs\"" Oct 30 17:29:43.843092 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:43.843075 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-27567a\"/\"default-dockercfg-nc6f8\"" Oct 30 17:29:43.943447 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:43.943410 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed4f3732-f99f-4218-a257-f66210947745-dshm\") pod \"distinct-w0-27567a-kserve-766b7789cc-6cnd7\" (UID: \"ed4f3732-f99f-4218-a257-f66210947745\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:29:43.943625 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:43.943467 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed4f3732-f99f-4218-a257-f66210947745-tls-certs\") pod \"distinct-w0-27567a-kserve-766b7789cc-6cnd7\" (UID: \"ed4f3732-f99f-4218-a257-f66210947745\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:29:43.943625 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:43.943584 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed4f3732-f99f-4218-a257-f66210947745-tmp-dir\") pod \"distinct-w0-27567a-kserve-766b7789cc-6cnd7\" (UID: \"ed4f3732-f99f-4218-a257-f66210947745\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:29:43.943738 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:43.943631 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed4f3732-f99f-4218-a257-f66210947745-model-cache\") pod \"distinct-w0-27567a-kserve-766b7789cc-6cnd7\" (UID: \"ed4f3732-f99f-4218-a257-f66210947745\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:29:43.943797 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:43.943776 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed4f3732-f99f-4218-a257-f66210947745-home\") pod \"distinct-w0-27567a-kserve-766b7789cc-6cnd7\" (UID: \"ed4f3732-f99f-4218-a257-f66210947745\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:29:43.943855 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:43.943809 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krjhb\" (UniqueName: \"kubernetes.io/projected/ed4f3732-f99f-4218-a257-f66210947745-kube-api-access-krjhb\") pod \"distinct-w0-27567a-kserve-766b7789cc-6cnd7\" (UID: \"ed4f3732-f99f-4218-a257-f66210947745\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:29:44.054967 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:44.054925 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed4f3732-f99f-4218-a257-f66210947745-dshm\") pod \"distinct-w0-27567a-kserve-766b7789cc-6cnd7\" (UID: \"ed4f3732-f99f-4218-a257-f66210947745\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:29:44.055297 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:44.055273 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed4f3732-f99f-4218-a257-f66210947745-tls-certs\") pod \"distinct-w0-27567a-kserve-766b7789cc-6cnd7\" (UID: \"ed4f3732-f99f-4218-a257-f66210947745\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:29:44.055451 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:44.055438 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed4f3732-f99f-4218-a257-f66210947745-tmp-dir\") pod \"distinct-w0-27567a-kserve-766b7789cc-6cnd7\" (UID: \"ed4f3732-f99f-4218-a257-f66210947745\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:29:44.055594 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:44.055575 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed4f3732-f99f-4218-a257-f66210947745-model-cache\") pod \"distinct-w0-27567a-kserve-766b7789cc-6cnd7\" (UID: \"ed4f3732-f99f-4218-a257-f66210947745\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:29:44.055770 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:44.055754 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed4f3732-f99f-4218-a257-f66210947745-home\") pod \"distinct-w0-27567a-kserve-766b7789cc-6cnd7\" (UID: \"ed4f3732-f99f-4218-a257-f66210947745\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:29:44.055885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:44.055853 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krjhb\" (UniqueName: \"kubernetes.io/projected/ed4f3732-f99f-4218-a257-f66210947745-kube-api-access-krjhb\") pod \"distinct-w0-27567a-kserve-766b7789cc-6cnd7\" (UID: \"ed4f3732-f99f-4218-a257-f66210947745\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:29:44.069405 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:44.069326 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed4f3732-f99f-4218-a257-f66210947745-tmp-dir\") pod \"distinct-w0-27567a-kserve-766b7789cc-6cnd7\" (UID: \"ed4f3732-f99f-4218-a257-f66210947745\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:29:44.070979 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:44.070947 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed4f3732-f99f-4218-a257-f66210947745-model-cache\") pod \"distinct-w0-27567a-kserve-766b7789cc-6cnd7\" (UID: \"ed4f3732-f99f-4218-a257-f66210947745\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:29:44.071539 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:44.071475 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed4f3732-f99f-4218-a257-f66210947745-home\") pod \"distinct-w0-27567a-kserve-766b7789cc-6cnd7\" (UID: \"ed4f3732-f99f-4218-a257-f66210947745\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:29:44.094897 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:44.088578 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed4f3732-f99f-4218-a257-f66210947745-tls-certs\") pod \"distinct-w0-27567a-kserve-766b7789cc-6cnd7\" (UID: \"ed4f3732-f99f-4218-a257-f66210947745\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:29:44.096260 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:44.096227 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed4f3732-f99f-4218-a257-f66210947745-dshm\") pod \"distinct-w0-27567a-kserve-766b7789cc-6cnd7\" (UID: \"ed4f3732-f99f-4218-a257-f66210947745\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:29:44.111274 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:44.111207 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krjhb\" (UniqueName: \"kubernetes.io/projected/ed4f3732-f99f-4218-a257-f66210947745-kube-api-access-krjhb\") pod \"distinct-w0-27567a-kserve-766b7789cc-6cnd7\" (UID: \"ed4f3732-f99f-4218-a257-f66210947745\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:29:44.132393 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:44.132366 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:29:44.136643 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:44.136622 4965 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 17:29:44.136705 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:44.136663 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" podUID="3f4d48ca-a065-4a13-89a7-efcc2301046f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 17:29:44.454181 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:44.454112 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7"] Oct 30 17:29:44.454564 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:29:44.454536 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded4f3732_f99f_4218_a257_f66210947745.slice/crio-7fb089398415d3105c13d9f249295a655ec0496058e513cc7278be6ea165f33a WatchSource:0}: Error finding container 7fb089398415d3105c13d9f249295a655ec0496058e513cc7278be6ea165f33a: Status 404 returned error can't find the container with id 7fb089398415d3105c13d9f249295a655ec0496058e513cc7278be6ea165f33a Oct 30 17:29:45.453062 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:45.453015 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" event={"ID":"ed4f3732-f99f-4218-a257-f66210947745","Type":"ContainerStarted","Data":"a372cb5a6e6913b84b9f45d65bc3b622f669dda8eddf35f35d2135d1cacc9ab9"} Oct 30 17:29:45.453489 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:45.453072 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" event={"ID":"ed4f3732-f99f-4218-a257-f66210947745","Type":"ContainerStarted","Data":"7fb089398415d3105c13d9f249295a655ec0496058e513cc7278be6ea165f33a"} Oct 30 17:29:45.500016 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:45.499980 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:45.500382 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:45.500354 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:45.501498 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:45.501476 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:29:45.528348 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:45.528296 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podStartSLOduration=2.528282911 podStartE2EDuration="2.528282911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:45.52644729 +0000 UTC m=+1043.578547265" watchObservedRunningTime="2026-10-30 17:29:45.528282911 +0000 UTC m=+1043.580382841" Oct 30 17:29:46.133454 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:46.133426 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:29:49.153363 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:49.153321 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:29:49.166326 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:49.166298 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:29:49.499988 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:49.499903 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:29:54.133100 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:54.133058 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:29:54.146986 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:54.146961 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:29:54.551676 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:29:54.551589 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:30:00.748251 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:00.748213 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:30:03.931834 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:03.931793 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp"] Oct 30 17:30:03.938467 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:03.938437 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:30:03.960691 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:03.960656 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp"] Oct 30 17:30:03.962653 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:03.962623 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-455086\"/\"distinct-2-w1-455086-kserve-self-signed-certs\"" Oct 30 17:30:03.978537 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:03.978499 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-home\") pod \"distinct-2-w1-455086-kserve-5c574bf567-nv4kp\" (UID: \"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d\") " pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:30:03.978537 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:03.978549 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-dshm\") pod \"distinct-2-w1-455086-kserve-5c574bf567-nv4kp\" (UID: \"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d\") " pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:30:03.978743 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:03.978657 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-tmp-dir\") pod \"distinct-2-w1-455086-kserve-5c574bf567-nv4kp\" (UID: \"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d\") " pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:30:03.978743 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:03.978693 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-model-cache\") pod \"distinct-2-w1-455086-kserve-5c574bf567-nv4kp\" (UID: \"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d\") " pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:30:03.978848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:03.978742 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8dq8\" (UniqueName: \"kubernetes.io/projected/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-kube-api-access-z8dq8\") pod \"distinct-2-w1-455086-kserve-5c574bf567-nv4kp\" (UID: \"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d\") " pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:30:03.978936 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:03.978918 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-tls-certs\") pod \"distinct-2-w1-455086-kserve-5c574bf567-nv4kp\" (UID: \"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d\") " pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:30:04.080064 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:04.080026 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-tls-certs\") pod \"distinct-2-w1-455086-kserve-5c574bf567-nv4kp\" (UID: \"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d\") " pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:30:04.080231 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:04.080164 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-home\") pod \"distinct-2-w1-455086-kserve-5c574bf567-nv4kp\" (UID: \"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d\") " pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:30:04.080231 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:04.080192 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-dshm\") pod \"distinct-2-w1-455086-kserve-5c574bf567-nv4kp\" (UID: \"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d\") " pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:30:04.080231 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:04.080226 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-tmp-dir\") pod \"distinct-2-w1-455086-kserve-5c574bf567-nv4kp\" (UID: \"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d\") " pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:30:04.080354 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:04.080252 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-model-cache\") pod \"distinct-2-w1-455086-kserve-5c574bf567-nv4kp\" (UID: \"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d\") " pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:30:04.080354 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:04.080306 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8dq8\" (UniqueName: \"kubernetes.io/projected/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-kube-api-access-z8dq8\") pod \"distinct-2-w1-455086-kserve-5c574bf567-nv4kp\" (UID: \"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d\") " pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:30:04.080615 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:04.080580 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-home\") pod \"distinct-2-w1-455086-kserve-5c574bf567-nv4kp\" (UID: \"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d\") " pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:30:04.080748 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:04.080636 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-tmp-dir\") pod \"distinct-2-w1-455086-kserve-5c574bf567-nv4kp\" (UID: \"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d\") " pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:30:04.080748 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:04.080722 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-model-cache\") pod \"distinct-2-w1-455086-kserve-5c574bf567-nv4kp\" (UID: \"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d\") " pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:30:04.082489 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:04.082467 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-dshm\") pod \"distinct-2-w1-455086-kserve-5c574bf567-nv4kp\" (UID: \"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d\") " pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:30:04.082730 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:04.082708 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-tls-certs\") pod \"distinct-2-w1-455086-kserve-5c574bf567-nv4kp\" (UID: \"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d\") " pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:30:04.103475 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:04.103450 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8dq8\" (UniqueName: \"kubernetes.io/projected/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-kube-api-access-z8dq8\") pod \"distinct-2-w1-455086-kserve-5c574bf567-nv4kp\" (UID: \"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d\") " pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:30:04.259929 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:04.259823 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:30:04.572991 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:04.572961 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp"] Oct 30 17:30:04.573777 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:30:04.573753 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38c90d17_2efa_4b7c_b10f_06a3a8b2a13d.slice/crio-9dd73286f8c67b9bafa63961b2bbcaaeccbcf3c3745b4deb029faa4a304eac0e WatchSource:0}: Error finding container 9dd73286f8c67b9bafa63961b2bbcaaeccbcf3c3745b4deb029faa4a304eac0e: Status 404 returned error can't find the container with id 9dd73286f8c67b9bafa63961b2bbcaaeccbcf3c3745b4deb029faa4a304eac0e Oct 30 17:30:04.616112 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:04.616082 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" event={"ID":"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d","Type":"ContainerStarted","Data":"9dd73286f8c67b9bafa63961b2bbcaaeccbcf3c3745b4deb029faa4a304eac0e"} Oct 30 17:30:05.623990 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:05.623952 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" event={"ID":"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d","Type":"ContainerStarted","Data":"de3f8588bd9afe41c9bde81c2e70045c698e7c6102bc9070fb2f868b35df7887"} Oct 30 17:30:05.667011 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:05.666918 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podStartSLOduration=2.666903048 podStartE2EDuration="2.666903048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:05.666081346 +0000 UTC m=+1063.718181271" watchObservedRunningTime="2026-10-30 17:30:05.666903048 +0000 UTC m=+1063.719002978" Oct 30 17:30:06.002044 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:30:06.001956 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:30:06.260158 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:06.260076 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:30:07.674992 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:07.674956 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp"] Oct 30 17:30:07.679850 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:07.679828 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:07.700434 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:07.700404 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp"] Oct 30 17:30:07.749890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:07.747683 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-credential-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:07.749890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:07.747742 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-istio-envoy\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:07.749890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:07.747825 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-workload-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:07.749890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:07.747905 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e4b4d14-b581-4844-a80e-a0edd473599e-istio-token\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:07.786770 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:07.786732 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-workload-certs\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:07.787093 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:07.787073 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e4b4d14-b581-4844-a80e-a0edd473599e-istio-podinfo\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:07.787674 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:07.787234 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrrpv\" (UniqueName: \"kubernetes.io/projected/5e4b4d14-b581-4844-a80e-a0edd473599e-kube-api-access-hrrpv\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:07.787674 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:07.787423 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e4b4d14-b581-4844-a80e-a0edd473599e-istiod-ca-cert\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:07.787674 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:07.787459 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-istio-data\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:07.888874 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:07.888829 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e4b4d14-b581-4844-a80e-a0edd473599e-istiod-ca-cert\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:07.889045 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:07.888884 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-istio-data\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:07.889045 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:07.888966 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-credential-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:07.889045 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:07.888989 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-istio-envoy\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:07.889045 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:07.889028 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-workload-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:07.889236 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:07.889069 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e4b4d14-b581-4844-a80e-a0edd473599e-istio-token\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:07.889236 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:07.889113 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-workload-certs\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:07.889236 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:07.889145 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e4b4d14-b581-4844-a80e-a0edd473599e-istio-podinfo\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:07.889236 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:07.889171 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrrpv\" (UniqueName: \"kubernetes.io/projected/5e4b4d14-b581-4844-a80e-a0edd473599e-kube-api-access-hrrpv\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:07.889503 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:07.889361 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-credential-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:07.889503 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:07.889440 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-istio-data\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:07.889605 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:07.889496 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-workload-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:07.889656 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:07.889618 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-workload-certs\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:07.889711 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:07.889686 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e4b4d14-b581-4844-a80e-a0edd473599e-istiod-ca-cert\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:07.891432 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:07.891409 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-istio-envoy\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:07.891730 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:07.891707 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e4b4d14-b581-4844-a80e-a0edd473599e-istio-podinfo\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:07.912436 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:07.912410 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrrpv\" (UniqueName: \"kubernetes.io/projected/5e4b4d14-b581-4844-a80e-a0edd473599e-kube-api-access-hrrpv\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:07.912566 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:07.912414 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e4b4d14-b581-4844-a80e-a0edd473599e-istio-token\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:07.995659 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:07.995578 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:08.317140 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:08.317109 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp"] Oct 30 17:30:08.317925 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:30:08.317896 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e4b4d14_b581_4844_a80e_a0edd473599e.slice/crio-53d588643f66139aedfb674d71910389e2b1ec123bc4374eb275abec1d99319b WatchSource:0}: Error finding container 53d588643f66139aedfb674d71910389e2b1ec123bc4374eb275abec1d99319b: Status 404 returned error can't find the container with id 53d588643f66139aedfb674d71910389e2b1ec123bc4374eb275abec1d99319b Oct 30 17:30:08.320298 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:08.320259 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:30:08.320411 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:08.320340 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:30:08.320411 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:08.320386 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:30:08.644186 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:08.644145 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" event={"ID":"5e4b4d14-b581-4844-a80e-a0edd473599e","Type":"ContainerStarted","Data":"0a4a72091f5c196bcdd521bfd19c5a7cdae4adbec5da81bc8389c2f65fda8f72"} Oct 30 17:30:08.644382 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:08.644215 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" event={"ID":"5e4b4d14-b581-4844-a80e-a0edd473599e","Type":"ContainerStarted","Data":"53d588643f66139aedfb674d71910389e2b1ec123bc4374eb275abec1d99319b"} Oct 30 17:30:08.697186 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:08.697009 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" podStartSLOduration=1.696989477 podStartE2EDuration="1.696989477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:08.695435749 +0000 UTC m=+1066.747535702" watchObservedRunningTime="2026-10-30 17:30:08.696989477 +0000 UTC m=+1066.749089410" Oct 30 17:30:08.996022 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:08.995927 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:09.997999 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:09.997966 4965 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 17:30:09.998427 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:09.998018 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" podUID="5e4b4d14-b581-4844-a80e-a0edd473599e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 17:30:10.996418 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:10.996381 4965 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 17:30:10.996576 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:10.996444 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" podUID="5e4b4d14-b581-4844-a80e-a0edd473599e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 17:30:12.000897 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:12.000041 4965 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 17:30:12.000897 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:12.000124 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" podUID="5e4b4d14-b581-4844-a80e-a0edd473599e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 17:30:12.996495 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:12.996459 4965 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 17:30:12.996678 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:12.996517 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" podUID="5e4b4d14-b581-4844-a80e-a0edd473599e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 17:30:13.996539 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:13.996502 4965 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 17:30:13.996963 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:13.996567 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" podUID="5e4b4d14-b581-4844-a80e-a0edd473599e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 17:30:14.260845 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:14.260750 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:30:14.274109 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:14.274074 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:30:14.719581 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:14.719554 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:30:14.866378 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:14.866339 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-phxks-zhmxv"] Oct 30 17:30:14.872348 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:14.872324 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-phxks-zhmxv" Oct 30 17:30:14.891626 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:14.891595 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-phxks-zhmxv"] Oct 30 17:30:14.996948 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:14.996854 4965 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 17:30:14.996948 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:14.996933 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" podUID="5e4b4d14-b581-4844-a80e-a0edd473599e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 17:30:15.003924 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:15.003893 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65150c77-1df1-42d6-b0f5-21ba7e248b59-service-ca-bundle\") pod \"periodic-gathering-phxks-zhmxv\" (UID: \"65150c77-1df1-42d6-b0f5-21ba7e248b59\") " pod="openshift-insights/periodic-gathering-phxks-zhmxv" Oct 30 17:30:15.004147 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:15.004122 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57cz9\" (UniqueName: \"kubernetes.io/projected/65150c77-1df1-42d6-b0f5-21ba7e248b59-kube-api-access-57cz9\") pod \"periodic-gathering-phxks-zhmxv\" (UID: \"65150c77-1df1-42d6-b0f5-21ba7e248b59\") " pod="openshift-insights/periodic-gathering-phxks-zhmxv" Oct 30 17:30:15.004260 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:15.004235 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/65150c77-1df1-42d6-b0f5-21ba7e248b59-archives-path\") pod \"periodic-gathering-phxks-zhmxv\" (UID: \"65150c77-1df1-42d6-b0f5-21ba7e248b59\") " pod="openshift-insights/periodic-gathering-phxks-zhmxv" Oct 30 17:30:15.004358 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:15.004339 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65150c77-1df1-42d6-b0f5-21ba7e248b59-trusted-ca-bundle\") pod \"periodic-gathering-phxks-zhmxv\" (UID: \"65150c77-1df1-42d6-b0f5-21ba7e248b59\") " pod="openshift-insights/periodic-gathering-phxks-zhmxv" Oct 30 17:30:15.105580 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:15.105546 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57cz9\" (UniqueName: \"kubernetes.io/projected/65150c77-1df1-42d6-b0f5-21ba7e248b59-kube-api-access-57cz9\") pod \"periodic-gathering-phxks-zhmxv\" (UID: \"65150c77-1df1-42d6-b0f5-21ba7e248b59\") " pod="openshift-insights/periodic-gathering-phxks-zhmxv" Oct 30 17:30:15.105762 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:15.105657 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/65150c77-1df1-42d6-b0f5-21ba7e248b59-archives-path\") pod \"periodic-gathering-phxks-zhmxv\" (UID: \"65150c77-1df1-42d6-b0f5-21ba7e248b59\") " pod="openshift-insights/periodic-gathering-phxks-zhmxv" Oct 30 17:30:15.105762 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:15.105708 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65150c77-1df1-42d6-b0f5-21ba7e248b59-trusted-ca-bundle\") pod \"periodic-gathering-phxks-zhmxv\" (UID: \"65150c77-1df1-42d6-b0f5-21ba7e248b59\") " pod="openshift-insights/periodic-gathering-phxks-zhmxv" Oct 30 17:30:15.105762 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:15.105750 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65150c77-1df1-42d6-b0f5-21ba7e248b59-service-ca-bundle\") pod \"periodic-gathering-phxks-zhmxv\" (UID: \"65150c77-1df1-42d6-b0f5-21ba7e248b59\") " pod="openshift-insights/periodic-gathering-phxks-zhmxv" Oct 30 17:30:15.106197 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:15.106173 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/65150c77-1df1-42d6-b0f5-21ba7e248b59-archives-path\") pod \"periodic-gathering-phxks-zhmxv\" (UID: \"65150c77-1df1-42d6-b0f5-21ba7e248b59\") " pod="openshift-insights/periodic-gathering-phxks-zhmxv" Oct 30 17:30:15.106651 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:15.106624 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65150c77-1df1-42d6-b0f5-21ba7e248b59-trusted-ca-bundle\") pod \"periodic-gathering-phxks-zhmxv\" (UID: \"65150c77-1df1-42d6-b0f5-21ba7e248b59\") " pod="openshift-insights/periodic-gathering-phxks-zhmxv" Oct 30 17:30:15.106809 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:15.106791 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65150c77-1df1-42d6-b0f5-21ba7e248b59-service-ca-bundle\") pod \"periodic-gathering-phxks-zhmxv\" (UID: \"65150c77-1df1-42d6-b0f5-21ba7e248b59\") " pod="openshift-insights/periodic-gathering-phxks-zhmxv" Oct 30 17:30:15.134070 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:15.134041 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57cz9\" (UniqueName: \"kubernetes.io/projected/65150c77-1df1-42d6-b0f5-21ba7e248b59-kube-api-access-57cz9\") pod \"periodic-gathering-phxks-zhmxv\" (UID: \"65150c77-1df1-42d6-b0f5-21ba7e248b59\") " pod="openshift-insights/periodic-gathering-phxks-zhmxv" Oct 30 17:30:15.195224 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:15.195185 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-phxks-zhmxv" Oct 30 17:30:15.773246 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:15.770889 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-phxks-zhmxv"] Oct 30 17:30:16.004887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:16.004839 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:16.008882 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:16.006304 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:16.008882 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:16.007601 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:30:16.727894 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:16.727627 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-phxks-zhmxv" event={"ID":"65150c77-1df1-42d6-b0f5-21ba7e248b59","Type":"ContainerStarted","Data":"069a0e9f3394469d0f518189a118fd8e70c721f83b54fa9b8fda18d67474874a"} Oct 30 17:30:16.727894 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:16.727666 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-phxks-zhmxv" event={"ID":"65150c77-1df1-42d6-b0f5-21ba7e248b59","Type":"ContainerStarted","Data":"26faf166ecfcc2ea1f83c0dd06955f2463f14fcdc70b38d22a3aa976d449eefe"} Oct 30 17:30:16.772240 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:16.772165 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-phxks-zhmxv" podStartSLOduration=2.772145192 podStartE2EDuration="2.772145192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:16.768504674 +0000 UTC m=+1074.820604606" watchObservedRunningTime="2026-10-30 17:30:16.772145192 +0000 UTC m=+1074.824245128" Oct 30 17:30:32.136024 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:32.135943 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj"] Oct 30 17:30:32.141392 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:32.141369 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:30:32.159436 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:32.159407 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj"] Oct 30 17:30:32.165873 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:32.165834 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-27567a\"/\"distinct-2-w0-27567a-kserve-self-signed-certs\"" Oct 30 17:30:32.213193 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:32.213163 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgz5v\" (UniqueName: \"kubernetes.io/projected/2a3bb727-21cb-470a-9de6-ce748fcdddb1-kube-api-access-bgz5v\") pod \"distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj\" (UID: \"2a3bb727-21cb-470a-9de6-ce748fcdddb1\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:30:32.213358 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:32.213288 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a3bb727-21cb-470a-9de6-ce748fcdddb1-model-cache\") pod \"distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj\" (UID: \"2a3bb727-21cb-470a-9de6-ce748fcdddb1\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:30:32.213358 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:32.213349 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a3bb727-21cb-470a-9de6-ce748fcdddb1-home\") pod \"distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj\" (UID: \"2a3bb727-21cb-470a-9de6-ce748fcdddb1\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:30:32.213489 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:32.213475 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a3bb727-21cb-470a-9de6-ce748fcdddb1-tmp-dir\") pod \"distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj\" (UID: \"2a3bb727-21cb-470a-9de6-ce748fcdddb1\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:30:32.213523 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:32.213511 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3bb727-21cb-470a-9de6-ce748fcdddb1-tls-certs\") pod \"distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj\" (UID: \"2a3bb727-21cb-470a-9de6-ce748fcdddb1\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:30:32.213557 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:32.213528 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a3bb727-21cb-470a-9de6-ce748fcdddb1-dshm\") pod \"distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj\" (UID: \"2a3bb727-21cb-470a-9de6-ce748fcdddb1\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:30:32.315109 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:32.315067 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a3bb727-21cb-470a-9de6-ce748fcdddb1-model-cache\") pod \"distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj\" (UID: \"2a3bb727-21cb-470a-9de6-ce748fcdddb1\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:30:32.328108 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:32.328068 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a3bb727-21cb-470a-9de6-ce748fcdddb1-home\") pod \"distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj\" (UID: \"2a3bb727-21cb-470a-9de6-ce748fcdddb1\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:30:32.328790 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:32.316009 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a3bb727-21cb-470a-9de6-ce748fcdddb1-model-cache\") pod \"distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj\" (UID: \"2a3bb727-21cb-470a-9de6-ce748fcdddb1\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:30:32.335752 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:32.335519 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a3bb727-21cb-470a-9de6-ce748fcdddb1-tmp-dir\") pod \"distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj\" (UID: \"2a3bb727-21cb-470a-9de6-ce748fcdddb1\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:30:32.336021 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:32.336001 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3bb727-21cb-470a-9de6-ce748fcdddb1-tls-certs\") pod \"distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj\" (UID: \"2a3bb727-21cb-470a-9de6-ce748fcdddb1\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:30:32.341883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:32.341842 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a3bb727-21cb-470a-9de6-ce748fcdddb1-dshm\") pod \"distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj\" (UID: \"2a3bb727-21cb-470a-9de6-ce748fcdddb1\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:30:32.342212 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:32.342190 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgz5v\" (UniqueName: \"kubernetes.io/projected/2a3bb727-21cb-470a-9de6-ce748fcdddb1-kube-api-access-bgz5v\") pod \"distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj\" (UID: \"2a3bb727-21cb-470a-9de6-ce748fcdddb1\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:30:32.342456 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:32.336291 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a3bb727-21cb-470a-9de6-ce748fcdddb1-home\") pod \"distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj\" (UID: \"2a3bb727-21cb-470a-9de6-ce748fcdddb1\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:30:32.342555 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:32.336528 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a3bb727-21cb-470a-9de6-ce748fcdddb1-tmp-dir\") pod \"distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj\" (UID: \"2a3bb727-21cb-470a-9de6-ce748fcdddb1\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:30:32.364934 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:32.361660 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3bb727-21cb-470a-9de6-ce748fcdddb1-tls-certs\") pod \"distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj\" (UID: \"2a3bb727-21cb-470a-9de6-ce748fcdddb1\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:30:32.383350 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:32.383320 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a3bb727-21cb-470a-9de6-ce748fcdddb1-dshm\") pod \"distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj\" (UID: \"2a3bb727-21cb-470a-9de6-ce748fcdddb1\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:30:32.384319 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:32.384293 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgz5v\" (UniqueName: \"kubernetes.io/projected/2a3bb727-21cb-470a-9de6-ce748fcdddb1-kube-api-access-bgz5v\") pod \"distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj\" (UID: \"2a3bb727-21cb-470a-9de6-ce748fcdddb1\") " pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:30:32.459758 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:32.459683 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:30:32.831021 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:32.830920 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj"] Oct 30 17:30:32.837509 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:32.837479 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" event={"ID":"2a3bb727-21cb-470a-9de6-ce748fcdddb1","Type":"ContainerStarted","Data":"39735219d29a507f240bf5b85b045b482716aedce310c376f91aeba8a9c4665f"} Oct 30 17:30:33.175900 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:33.175831 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:30:33.844800 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:33.844758 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" event={"ID":"2a3bb727-21cb-470a-9de6-ce748fcdddb1","Type":"ContainerStarted","Data":"bcf371609a3f73ce984f0bd7cc308488882ace523cacea3213819145d261afa9"} Oct 30 17:30:33.887500 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:33.887448 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podStartSLOduration=1.88743617 podStartE2EDuration="1.88743617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:33.886034136 +0000 UTC m=+1091.938134064" watchObservedRunningTime="2026-10-30 17:30:33.88743617 +0000 UTC m=+1091.939536101" Oct 30 17:30:34.461063 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:34.461030 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:30:35.997047 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:30:35.997011 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:30:37.998030 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:37.997995 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw"] Oct 30 17:30:38.002430 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.002410 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:38.024017 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.023986 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm"] Oct 30 17:30:38.029363 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.029338 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:38.036518 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.036486 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw"] Oct 30 17:30:38.049189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.049158 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm"] Oct 30 17:30:38.084780 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.084733 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp"] Oct 30 17:30:38.089761 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.089735 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:38.114909 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.114882 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp"] Oct 30 17:30:38.132771 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.132743 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istio-data\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:38.132958 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.132823 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-istio-envoy\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:38.132958 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.132846 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d493c105-7ba4-4626-b438-2224db37a905-istiod-ca-cert\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:38.133101 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.133005 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-workload-certs\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:38.133101 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.133042 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istio-token\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:38.133101 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.133073 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-workload-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:38.133198 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.133185 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d493c105-7ba4-4626-b438-2224db37a905-istio-token\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:38.133235 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.133207 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istio-podinfo\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:38.133235 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.133229 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-istio-data\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:38.133297 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.133263 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-workload-certs\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:38.133297 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.133290 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-credential-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:38.133367 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.133312 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxmbb\" (UniqueName: \"kubernetes.io/projected/d493c105-7ba4-4626-b438-2224db37a905-kube-api-access-vxmbb\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:38.133367 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.133344 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d493c105-7ba4-4626-b438-2224db37a905-istio-podinfo\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:38.133367 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.133358 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-credential-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:38.133495 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.133377 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7l64\" (UniqueName: \"kubernetes.io/projected/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-kube-api-access-r7l64\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:38.133495 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.133404 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-workload-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:38.133495 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.133428 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istio-envoy\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:38.133623 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.133554 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istiod-ca-cert\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:38.235066 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.235036 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d493c105-7ba4-4626-b438-2224db37a905-istio-token\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:38.235066 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.235069 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istio-podinfo\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:38.235279 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.235091 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-istio-data\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:38.235279 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.235124 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-workload-certs\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:38.235279 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.235158 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c868037-59d9-480a-87c1-f29251d35284-istio-podinfo\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:38.235279 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.235180 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c868037-59d9-480a-87c1-f29251d35284-istio-token\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:38.235279 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.235261 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-credential-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:38.235547 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.235303 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxmbb\" (UniqueName: \"kubernetes.io/projected/d493c105-7ba4-4626-b438-2224db37a905-kube-api-access-vxmbb\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:38.235547 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.235355 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d493c105-7ba4-4626-b438-2224db37a905-istio-podinfo\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:38.235547 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.235381 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-credential-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:38.235547 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.235407 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7l64\" (UniqueName: \"kubernetes.io/projected/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-kube-api-access-r7l64\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:38.235547 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.235439 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-istio-envoy\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:38.235547 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.235472 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-workload-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:38.235547 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.235500 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-istio-data\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:38.235547 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.235504 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istio-envoy\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:38.236026 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.235575 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-workload-certs\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:38.236026 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.235606 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-istio-data\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:38.236026 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.235636 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c868037-59d9-480a-87c1-f29251d35284-istiod-ca-cert\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:38.236026 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.235645 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-credential-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:38.236026 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.235802 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istiod-ca-cert\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:38.236026 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.235832 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-credential-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:38.236026 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.235909 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istio-data\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:38.236026 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.235913 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-credential-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:38.236026 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.235957 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-workload-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:38.236026 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.235965 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-istio-envoy\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:38.236026 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.236017 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d493c105-7ba4-4626-b438-2224db37a905-istiod-ca-cert\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:38.236609 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.236060 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-workload-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:38.236609 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.236110 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtqr5\" (UniqueName: \"kubernetes.io/projected/5c868037-59d9-480a-87c1-f29251d35284-kube-api-access-rtqr5\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:38.236609 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.236179 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istio-data\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:38.236609 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.236183 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-workload-certs\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:38.236609 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.236224 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istio-token\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:38.236609 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.236257 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-workload-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:38.236609 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.236291 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-workload-certs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:38.236609 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.236456 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-workload-certs\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:38.236609 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.236472 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istiod-ca-cert\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:38.237094 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.236876 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-workload-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:38.237185 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.237162 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d493c105-7ba4-4626-b438-2224db37a905-istiod-ca-cert\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:38.238045 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.238025 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istio-envoy\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:38.238300 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.238280 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istio-podinfo\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:38.238459 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.238442 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-istio-envoy\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:38.238841 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.238823 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d493c105-7ba4-4626-b438-2224db37a905-istio-podinfo\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:38.258631 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.258565 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7l64\" (UniqueName: \"kubernetes.io/projected/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-kube-api-access-r7l64\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:38.258735 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.258664 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d493c105-7ba4-4626-b438-2224db37a905-istio-token\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:38.259053 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.259030 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxmbb\" (UniqueName: \"kubernetes.io/projected/d493c105-7ba4-4626-b438-2224db37a905-kube-api-access-vxmbb\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:38.259236 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.259220 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istio-token\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:38.319336 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.319310 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:38.337443 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.337420 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-credential-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:38.337549 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.337467 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-workload-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:38.337549 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.337496 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtqr5\" (UniqueName: \"kubernetes.io/projected/5c868037-59d9-480a-87c1-f29251d35284-kube-api-access-rtqr5\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:38.337642 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.337547 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-workload-certs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:38.337642 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.337623 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c868037-59d9-480a-87c1-f29251d35284-istio-podinfo\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:38.337744 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.337645 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c868037-59d9-480a-87c1-f29251d35284-istio-token\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:38.337744 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.337699 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-istio-envoy\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:38.337858 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.337782 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-istio-data\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:38.337858 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.337808 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c868037-59d9-480a-87c1-f29251d35284-istiod-ca-cert\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:38.337986 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.337940 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-credential-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:38.338154 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.338129 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-workload-certs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:38.338274 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.338169 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-workload-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:38.338414 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.338393 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-istio-data\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:38.338822 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.338801 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c868037-59d9-480a-87c1-f29251d35284-istiod-ca-cert\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:38.340854 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.340809 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c868037-59d9-480a-87c1-f29251d35284-istio-podinfo\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:38.340971 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.340911 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-istio-envoy\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:38.348917 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.348732 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:38.361047 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.361009 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c868037-59d9-480a-87c1-f29251d35284-istio-token\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:38.361191 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.361169 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtqr5\" (UniqueName: \"kubernetes.io/projected/5c868037-59d9-480a-87c1-f29251d35284-kube-api-access-rtqr5\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:38.417972 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.417755 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:38.644084 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.644051 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw"] Oct 30 17:30:38.644705 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:30:38.644677 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb6f0cf_bfbc_4b53_92f3_eac354a8d1a4.slice/crio-c65f2e4f2602b684ec777f63a23b6c75ac6451fde2b1b2196a34eeeebe2b1eb4 WatchSource:0}: Error finding container c65f2e4f2602b684ec777f63a23b6c75ac6451fde2b1b2196a34eeeebe2b1eb4: Status 404 returned error can't find the container with id c65f2e4f2602b684ec777f63a23b6c75ac6451fde2b1b2196a34eeeebe2b1eb4 Oct 30 17:30:38.647151 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.647117 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:30:38.647234 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.647177 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:30:38.647234 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.647203 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:30:38.690267 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.686644 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm"] Oct 30 17:30:38.690267 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:30:38.687248 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd493c105_7ba4_4626_b438_2224db37a905.slice/crio-87d27bd37225802032b9c6b32a994c9dc4386368244cc9bbbf0e55a92f7042cd WatchSource:0}: Error finding container 87d27bd37225802032b9c6b32a994c9dc4386368244cc9bbbf0e55a92f7042cd: Status 404 returned error can't find the container with id 87d27bd37225802032b9c6b32a994c9dc4386368244cc9bbbf0e55a92f7042cd Oct 30 17:30:38.691536 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.691495 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:30:38.691625 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.691581 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:30:38.691668 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.691623 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:30:38.744905 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.743879 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp"] Oct 30 17:30:38.744905 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:30:38.744576 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c868037_59d9_480a_87c1_f29251d35284.slice/crio-26c6ef068c69a0498b1f1366197d28e69a1d86b009f63cd09c8e69895cf05ce1 WatchSource:0}: Error finding container 26c6ef068c69a0498b1f1366197d28e69a1d86b009f63cd09c8e69895cf05ce1: Status 404 returned error can't find the container with id 26c6ef068c69a0498b1f1366197d28e69a1d86b009f63cd09c8e69895cf05ce1 Oct 30 17:30:38.747105 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.747073 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:30:38.747227 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.747137 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:30:38.747227 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.747171 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:30:38.889404 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.889287 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" event={"ID":"5c868037-59d9-480a-87c1-f29251d35284","Type":"ContainerStarted","Data":"499923eb9585afcca4e0c93b287ca973a017b039f0c24a4f56aee25e2ce88d67"} Oct 30 17:30:38.889404 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.889353 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" event={"ID":"5c868037-59d9-480a-87c1-f29251d35284","Type":"ContainerStarted","Data":"26c6ef068c69a0498b1f1366197d28e69a1d86b009f63cd09c8e69895cf05ce1"} Oct 30 17:30:38.892233 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.892163 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" event={"ID":"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4","Type":"ContainerStarted","Data":"41e8ee7eab6820251f5a672d3ab50d3f46b6ff5353cd4feae259e566d8ed2ea1"} Oct 30 17:30:38.895950 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.895929 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" event={"ID":"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4","Type":"ContainerStarted","Data":"c65f2e4f2602b684ec777f63a23b6c75ac6451fde2b1b2196a34eeeebe2b1eb4"} Oct 30 17:30:38.896141 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.896124 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" event={"ID":"d493c105-7ba4-4626-b438-2224db37a905","Type":"ContainerStarted","Data":"47ea6ede04b24658360eb73bdc317de52c07c1353ff256597cb85d778898878f"} Oct 30 17:30:38.896265 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:38.896249 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" event={"ID":"d493c105-7ba4-4626-b438-2224db37a905","Type":"ContainerStarted","Data":"87d27bd37225802032b9c6b32a994c9dc4386368244cc9bbbf0e55a92f7042cd"} Oct 30 17:30:39.000498 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:39.000431 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" podStartSLOduration=1.000412229 podStartE2EDuration="1.000412229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:38.952694932 +0000 UTC m=+1097.004794866" watchObservedRunningTime="2026-10-30 17:30:39.000412229 +0000 UTC m=+1097.052512156" Oct 30 17:30:39.001078 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:39.001039 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" podStartSLOduration=2.001024615 podStartE2EDuration="2.001024615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:38.997509726 +0000 UTC m=+1097.049609721" watchObservedRunningTime="2026-10-30 17:30:39.001024615 +0000 UTC m=+1097.053124566" Oct 30 17:30:39.045071 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:39.045009 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" podStartSLOduration=2.044989851 podStartE2EDuration="2.044989851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:39.042959946 +0000 UTC m=+1097.095059915" watchObservedRunningTime="2026-10-30 17:30:39.044989851 +0000 UTC m=+1097.097089783" Oct 30 17:30:39.333938 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:39.321771 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:39.333938 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:39.321968 4965 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 17:30:39.333938 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:39.322045 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" podUID="5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 17:30:39.349259 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:39.349131 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:39.354065 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:39.354031 4965 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 17:30:39.354358 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:39.354311 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" podUID="d493c105-7ba4-4626-b438-2224db37a905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 17:30:39.418930 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:39.418018 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:39.424072 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:39.424027 4965 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 17:30:39.424417 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:39.424388 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" podUID="5c868037-59d9-480a-87c1-f29251d35284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 17:30:40.332888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:40.332716 4965 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 17:30:40.332888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:40.332777 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" podUID="5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 17:30:40.349233 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:40.349195 4965 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 17:30:40.349401 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:40.349250 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" podUID="d493c105-7ba4-4626-b438-2224db37a905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 17:30:40.423888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:40.423298 4965 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 17:30:40.423888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:40.423362 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" podUID="5c868037-59d9-480a-87c1-f29251d35284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 17:30:41.331633 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:41.331016 4965 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 17:30:41.331633 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:41.331081 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" podUID="5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 17:30:41.350991 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:41.349844 4965 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 17:30:41.350991 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:41.349925 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" podUID="d493c105-7ba4-4626-b438-2224db37a905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 17:30:41.437911 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:41.433643 4965 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 17:30:41.437911 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:41.433701 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" podUID="5c868037-59d9-480a-87c1-f29251d35284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 17:30:42.323488 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:42.323446 4965 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 17:30:42.325963 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:42.325928 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" podUID="5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 17:30:42.350334 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:42.350006 4965 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 17:30:42.350334 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:42.350055 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" podUID="d493c105-7ba4-4626-b438-2224db37a905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 17:30:42.418245 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:42.418208 4965 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 17:30:42.418821 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:42.418793 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" podUID="5c868037-59d9-480a-87c1-f29251d35284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 17:30:42.479888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:42.473754 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:30:42.551887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:42.551744 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:30:43.248960 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:43.248933 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:30:43.336888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:43.328243 4965 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 17:30:43.336888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:43.328306 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" podUID="5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 17:30:43.350427 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:43.349550 4965 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 17:30:43.350427 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:43.349603 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" podUID="d493c105-7ba4-4626-b438-2224db37a905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 17:30:43.486898 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:43.455774 4965 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 17:30:43.486898 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:43.455836 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" podUID="5c868037-59d9-480a-87c1-f29251d35284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 17:30:44.321886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:44.321259 4965 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 17:30:44.321886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:44.321321 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" podUID="5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 17:30:44.349216 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:44.349178 4965 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 17:30:44.349467 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:44.349438 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" podUID="d493c105-7ba4-4626-b438-2224db37a905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 17:30:44.420356 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:44.420323 4965 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 17:30:44.420636 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:44.420609 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" podUID="5c868037-59d9-480a-87c1-f29251d35284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 17:30:45.321394 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:45.321342 4965 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 17:30:45.321922 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:45.321411 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" podUID="5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 17:30:45.349677 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:45.349642 4965 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 17:30:45.349887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:45.349747 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" podUID="d493c105-7ba4-4626-b438-2224db37a905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 17:30:45.418671 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:45.418632 4965 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 17:30:45.418920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:45.418695 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" podUID="5c868037-59d9-480a-87c1-f29251d35284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 17:30:46.320383 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:46.319617 4965 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 17:30:46.320383 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:46.319674 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" podUID="5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 17:30:46.349925 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:46.349066 4965 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 17:30:46.349925 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:46.349125 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" podUID="d493c105-7ba4-4626-b438-2224db37a905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 17:30:46.979080 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:46.979048 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:46.994271 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:46.994219 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:46.997794 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:46.995891 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:30:47.329965 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:47.328631 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:47.329965 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:47.329680 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:47.333922 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:47.330758 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:30:47.371798 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:47.367855 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:47.371798 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:47.368774 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:47.371798 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:47.369369 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:30:52.824615 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:52.824578 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5"] Oct 30 17:30:52.830630 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:52.830571 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:30:52.854149 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:52.854123 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5"] Oct 30 17:30:52.868700 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:52.868670 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm"] Oct 30 17:30:52.873932 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:52.873910 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:30:52.891600 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:52.891010 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm"] Oct 30 17:30:52.934811 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:52.934778 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-workload-certs\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:30:52.934991 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:52.934839 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-istio-data\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:30:52.934991 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:52.934929 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f46hz\" (UniqueName: \"kubernetes.io/projected/d8f96554-6aec-4052-8219-307fb11b6404-kube-api-access-f46hz\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:30:52.934991 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:52.934958 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-istio-envoy\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:30:52.934991 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:52.934986 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8f96554-6aec-4052-8219-307fb11b6404-istio-token\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:30:52.935140 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:52.935046 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8f96554-6aec-4052-8219-307fb11b6404-istiod-ca-cert\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:30:52.935175 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:52.935162 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-credential-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:30:52.935212 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:52.935200 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-workload-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:30:52.935318 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:52.935304 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8f96554-6aec-4052-8219-307fb11b6404-istio-podinfo\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:30:53.036953 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.036915 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-workload-certs\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:30:53.037130 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.036969 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-workload-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:30:53.037130 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.037017 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-istio-data\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:30:53.037130 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.037103 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f46hz\" (UniqueName: \"kubernetes.io/projected/d8f96554-6aec-4052-8219-307fb11b6404-kube-api-access-f46hz\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:30:53.037328 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.037216 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istiod-ca-cert\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:30:53.037328 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.037251 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-istio-envoy\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:30:53.037428 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.037328 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8f96554-6aec-4052-8219-307fb11b6404-istio-token\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:30:53.037428 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.037343 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-workload-certs\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:30:53.037428 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.037368 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istio-envoy\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:30:53.037428 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.037389 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-istio-data\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:30:53.037428 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.037409 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8f96554-6aec-4052-8219-307fb11b6404-istiod-ca-cert\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:30:53.037652 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.037469 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-credential-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:30:53.037652 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.037519 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-workload-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:30:53.037652 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.037622 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8f96554-6aec-4052-8219-307fb11b6404-istio-podinfo\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:30:53.037808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.037654 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istio-podinfo\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:30:53.037808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.037696 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-workload-certs\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:30:53.037808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.037748 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istio-data\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:30:53.037808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.037777 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-credential-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:30:53.038025 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.037809 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istio-token\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:30:53.038025 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.037857 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9n98\" (UniqueName: \"kubernetes.io/projected/c7db6ef5-c274-4ca7-b20b-e2eb17592720-kube-api-access-p9n98\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:30:53.038025 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.037929 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-credential-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:30:53.038187 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.038071 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-workload-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:30:53.038187 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.038172 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8f96554-6aec-4052-8219-307fb11b6404-istiod-ca-cert\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:30:53.039591 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.039564 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-istio-envoy\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:30:53.039940 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.039924 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8f96554-6aec-4052-8219-307fb11b6404-istio-podinfo\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:30:53.060606 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.060578 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8f96554-6aec-4052-8219-307fb11b6404-istio-token\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:30:53.060724 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.060701 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f46hz\" (UniqueName: \"kubernetes.io/projected/d8f96554-6aec-4052-8219-307fb11b6404-kube-api-access-f46hz\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:30:53.113163 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.113094 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq"] Oct 30 17:30:53.118353 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.118334 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:30:53.138232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.138207 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq"] Oct 30 17:30:53.138902 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.138879 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istio-podinfo\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:30:53.138987 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.138913 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-workload-certs\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:30:53.138987 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.138947 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istio-data\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:30:53.138987 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.138967 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-credential-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:30:53.139085 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.138988 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istio-token\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:30:53.139085 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.139018 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9n98\" (UniqueName: \"kubernetes.io/projected/c7db6ef5-c274-4ca7-b20b-e2eb17592720-kube-api-access-p9n98\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:30:53.139085 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.139079 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-workload-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:30:53.139171 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.139130 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istiod-ca-cert\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:30:53.139171 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.139165 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istio-envoy\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:30:53.140194 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.139829 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-credential-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:30:53.140194 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.140083 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-workload-certs\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:30:53.140194 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.140088 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istio-data\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:30:53.140390 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.140297 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-workload-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:30:53.140390 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.140315 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istiod-ca-cert\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:30:53.142749 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.142727 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istio-podinfo\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:30:53.145118 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.145094 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istio-envoy\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:30:53.145331 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.145313 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:30:53.163464 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.163420 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istio-token\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:30:53.163619 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.163569 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9n98\" (UniqueName: \"kubernetes.io/projected/c7db6ef5-c274-4ca7-b20b-e2eb17592720-kube-api-access-p9n98\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:30:53.194140 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.194110 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:30:53.241584 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.241544 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-istio-data\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:30:53.241746 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.241587 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvctd\" (UniqueName: \"kubernetes.io/projected/9f45f733-1397-4de5-81fd-b035baef4fb4-kube-api-access-jvctd\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:30:53.241746 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.241664 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-istio-envoy\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:30:53.241746 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.241710 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f45f733-1397-4de5-81fd-b035baef4fb4-istio-podinfo\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:30:53.241746 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.241734 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f45f733-1397-4de5-81fd-b035baef4fb4-istiod-ca-cert\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:30:53.241990 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.241846 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f45f733-1397-4de5-81fd-b035baef4fb4-istio-token\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:30:53.241990 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.241979 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-workload-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:30:53.242120 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.242103 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-workload-certs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:30:53.242222 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.242209 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-credential-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:30:53.344199 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.344163 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-workload-certs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:30:53.344396 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.344252 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-credential-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:30:53.344396 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.344310 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-istio-data\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:30:53.344396 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.344341 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvctd\" (UniqueName: \"kubernetes.io/projected/9f45f733-1397-4de5-81fd-b035baef4fb4-kube-api-access-jvctd\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:30:53.344589 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.344422 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-istio-envoy\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:30:53.344589 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.344474 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f45f733-1397-4de5-81fd-b035baef4fb4-istio-podinfo\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:30:53.344589 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.344499 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f45f733-1397-4de5-81fd-b035baef4fb4-istiod-ca-cert\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:30:53.344753 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.344581 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-workload-certs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:30:53.344753 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.344597 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f45f733-1397-4de5-81fd-b035baef4fb4-istio-token\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:30:53.344753 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.344703 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-workload-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:30:53.345094 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.345074 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-workload-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:30:53.345200 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.345178 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-credential-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:30:53.345693 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.345644 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-istio-data\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:30:53.346204 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.346179 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f45f733-1397-4de5-81fd-b035baef4fb4-istiod-ca-cert\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:30:53.347797 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.347773 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-istio-envoy\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:30:53.348646 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.348620 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f45f733-1397-4de5-81fd-b035baef4fb4-istio-podinfo\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:30:53.367453 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.367399 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f45f733-1397-4de5-81fd-b035baef4fb4-istio-token\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:30:53.368038 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.368020 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvctd\" (UniqueName: \"kubernetes.io/projected/9f45f733-1397-4de5-81fd-b035baef4fb4-kube-api-access-jvctd\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:30:53.432795 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.432765 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:30:53.473412 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:30:53.473382 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8f96554_6aec_4052_8219_307fb11b6404.slice/crio-00e4fb6dcb3e40e2761b4ef50b529ced3d2be7400034676173fde16b1664fca4 WatchSource:0}: Error finding container 00e4fb6dcb3e40e2761b4ef50b529ced3d2be7400034676173fde16b1664fca4: Status 404 returned error can't find the container with id 00e4fb6dcb3e40e2761b4ef50b529ced3d2be7400034676173fde16b1664fca4 Oct 30 17:30:53.473531 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.473492 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5"] Oct 30 17:30:53.476119 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.476077 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:30:53.476233 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.476166 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:30:53.476233 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.476209 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:30:53.522218 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.522176 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm"] Oct 30 17:30:53.527532 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:30:53.527498 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7db6ef5_c274_4ca7_b20b_e2eb17592720.slice/crio-f189795347da6d93b4963c62ff76189cc2551d990416aad001f498edd0f59071 WatchSource:0}: Error finding container f189795347da6d93b4963c62ff76189cc2551d990416aad001f498edd0f59071: Status 404 returned error can't find the container with id f189795347da6d93b4963c62ff76189cc2551d990416aad001f498edd0f59071 Oct 30 17:30:53.531310 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.531277 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:30:53.531420 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.531347 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:30:53.531420 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.531378 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:30:53.746259 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.746232 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq"] Oct 30 17:30:53.747082 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:30:53.747053 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f45f733_1397_4de5_81fd_b035baef4fb4.slice/crio-aa3e5931457e4f98ebb87624d65b55740042f62c258de07e5eaf4a60499ea0db WatchSource:0}: Error finding container aa3e5931457e4f98ebb87624d65b55740042f62c258de07e5eaf4a60499ea0db: Status 404 returned error can't find the container with id aa3e5931457e4f98ebb87624d65b55740042f62c258de07e5eaf4a60499ea0db Oct 30 17:30:53.749771 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.749733 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:30:53.749903 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.749799 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:30:53.749903 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:53.749825 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:30:54.430677 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:54.430639 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" event={"ID":"9f45f733-1397-4de5-81fd-b035baef4fb4","Type":"ContainerStarted","Data":"c55a6ba88b03e73fcc59835e1762a6237e0371c5e89c93dbe498707dda12691f"} Oct 30 17:30:54.431342 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:54.431314 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" event={"ID":"9f45f733-1397-4de5-81fd-b035baef4fb4","Type":"ContainerStarted","Data":"aa3e5931457e4f98ebb87624d65b55740042f62c258de07e5eaf4a60499ea0db"} Oct 30 17:30:54.436043 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:54.435935 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:30:54.438300 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:54.436175 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" event={"ID":"c7db6ef5-c274-4ca7-b20b-e2eb17592720","Type":"ContainerStarted","Data":"67de36f2cc0a263f94c43e8d5cae63bb64585f66bb0f66d71f1772d011ec2f30"} Oct 30 17:30:54.438300 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:54.436216 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" event={"ID":"c7db6ef5-c274-4ca7-b20b-e2eb17592720","Type":"ContainerStarted","Data":"f189795347da6d93b4963c62ff76189cc2551d990416aad001f498edd0f59071"} Oct 30 17:30:54.438300 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:54.438060 4965 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 17:30:54.438300 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:54.438113 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" podUID="9f45f733-1397-4de5-81fd-b035baef4fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 17:30:54.440668 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:54.440630 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" event={"ID":"d8f96554-6aec-4052-8219-307fb11b6404","Type":"ContainerStarted","Data":"9c65aa992d826ddf4bd112dc532b7cded0a020ecf963add7b2cb0d8d69d2ce3d"} Oct 30 17:30:54.441693 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:54.441671 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" event={"ID":"d8f96554-6aec-4052-8219-307fb11b6404","Type":"ContainerStarted","Data":"00e4fb6dcb3e40e2761b4ef50b529ced3d2be7400034676173fde16b1664fca4"} Oct 30 17:30:54.534917 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:54.533348 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" podStartSLOduration=1.533327403 podStartE2EDuration="1.533327403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:54.48231226 +0000 UTC m=+1112.534412191" watchObservedRunningTime="2026-10-30 17:30:54.533327403 +0000 UTC m=+1112.585427335" Oct 30 17:30:54.639920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:54.588833 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" podStartSLOduration=2.588810847 podStartE2EDuration="2.588810847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:54.585406581 +0000 UTC m=+1112.637506514" watchObservedRunningTime="2026-10-30 17:30:54.588810847 +0000 UTC m=+1112.640910780" Oct 30 17:30:54.639920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:54.639667 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" podStartSLOduration=2.639651134 podStartE2EDuration="2.639651134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:54.540230055 +0000 UTC m=+1112.592330011" watchObservedRunningTime="2026-10-30 17:30:54.639651134 +0000 UTC m=+1112.691751066" Oct 30 17:30:55.146026 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:55.145992 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:30:55.147318 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:55.147292 4965 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 17:30:55.147428 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:55.147336 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" podUID="d8f96554-6aec-4052-8219-307fb11b6404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 17:30:55.198248 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:55.198211 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:30:55.201000 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:55.200968 4965 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 17:30:55.207911 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:55.201178 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" podUID="c7db6ef5-c274-4ca7-b20b-e2eb17592720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 17:30:55.433386 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:55.433291 4965 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 17:30:55.433980 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:55.433946 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" podUID="9f45f733-1397-4de5-81fd-b035baef4fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 17:30:56.146844 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:56.146807 4965 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 17:30:56.147046 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:56.146894 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" podUID="d8f96554-6aec-4052-8219-307fb11b6404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 17:30:56.201893 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:56.197207 4965 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 17:30:56.201893 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:56.197268 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" podUID="c7db6ef5-c274-4ca7-b20b-e2eb17592720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 17:30:56.439305 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:56.438290 4965 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 17:30:56.439305 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:56.438357 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" podUID="9f45f733-1397-4de5-81fd-b035baef4fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 17:30:57.147231 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:57.147189 4965 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 17:30:57.147410 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:57.147256 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" podUID="d8f96554-6aec-4052-8219-307fb11b6404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 17:30:57.198888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:57.195345 4965 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 17:30:57.198888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:57.195403 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" podUID="c7db6ef5-c274-4ca7-b20b-e2eb17592720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 17:30:57.447968 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:57.441221 4965 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 17:30:57.447968 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:57.441286 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" podUID="9f45f733-1397-4de5-81fd-b035baef4fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 17:30:58.157887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:58.156957 4965 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 17:30:58.157887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:58.157111 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" podUID="d8f96554-6aec-4052-8219-307fb11b6404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 17:30:58.203889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:58.197202 4965 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 17:30:58.203889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:58.197262 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" podUID="c7db6ef5-c274-4ca7-b20b-e2eb17592720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 17:30:58.436950 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:58.435307 4965 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 17:30:58.436950 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:58.435372 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" podUID="9f45f733-1397-4de5-81fd-b035baef4fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 17:30:59.146312 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:59.146278 4965 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 17:30:59.146716 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:59.146327 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" podUID="d8f96554-6aec-4052-8219-307fb11b6404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 17:30:59.198887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:59.197799 4965 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 17:30:59.198887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:59.197888 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" podUID="c7db6ef5-c274-4ca7-b20b-e2eb17592720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 17:30:59.435566 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:59.435249 4965 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 17:30:59.435566 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:30:59.435321 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" podUID="9f45f733-1397-4de5-81fd-b035baef4fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 17:31:00.145822 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:00.145786 4965 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 17:31:00.146066 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:00.145841 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" podUID="d8f96554-6aec-4052-8219-307fb11b6404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 17:31:00.195106 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:00.195072 4965 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 17:31:00.195672 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:00.195638 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" podUID="c7db6ef5-c274-4ca7-b20b-e2eb17592720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 17:31:00.433890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:00.433779 4965 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 17:31:00.434069 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:00.433854 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" podUID="9f45f733-1397-4de5-81fd-b035baef4fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 17:31:01.463220 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:01.463065 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:31:01.465526 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:01.465503 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:31:01.467238 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:01.467215 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:31:01.593159 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:01.593125 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:31:01.594344 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:01.594319 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:31:01.595824 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:01.595804 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 17:31:01.675836 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:01.675803 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:31:01.677129 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:01.677105 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:31:01.678208 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:01.678188 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:31:02.150356 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:02.150316 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:31:05.141942 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:05.141052 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded" Oct 30 17:31:05.195887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:05.194977 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:05.272113 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:05.272076 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:05.456069 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:05.455959 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:05.502036 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:05.501991 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:05.554024 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:05.553974 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:05.555632 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:05.554333 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 17:31:05.634994 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:05.634946 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:05.703003 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:05.702956 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 17:31:06.304938 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:06.301637 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:06.559615 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:06.559513 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 17:31:06.616989 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:06.563906 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 17:31:06.616989 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:06.613745 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 17:31:06.648341 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:06.648296 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 17:31:06.653211 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:06.653170 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 17:31:06.708511 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:06.708170 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:06.756889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:06.735364 4965 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:31:06.756889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:06.735460 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" podUID="16d015c9-b3d7-45ae-a4f7-04aa68f678fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:07.288898 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:31:07.270228 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:31:07.583916 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:31:07.559638 4965 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.456s" Oct 30 17:31:07.647979 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:07.647951 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:31:07.688022 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:07.687993 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:31:07.689169 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:07.689147 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:31:07.728997 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:07.621118 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:07.729325 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:07.647803 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:07.739889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:07.560351 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:07.755888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:07.649469 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:07.940431 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:07.655932 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:07.952883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:07.564944 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:07.952883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:07.590014 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs" podUID="d8d199b0-9f65-4d32-aafb-62e8dc216076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:07.952883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:07.733898 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:31:07.952883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:07.949351 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:31:07.952883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:07.949392 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:31:07.952883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:07.949445 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:31:07.952883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:07.949467 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc"] Oct 30 17:31:07.959101 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:07.756130 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 17:31:07.990104 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:07.967032 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:31:07.990104 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:07.967067 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:31:07.990104 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:07.967143 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:07.990104 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:07.972450 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:31:08.017063 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.008267 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:31:08.017063 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.012657 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:31:08.032953 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.030494 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc"] Oct 30 17:31:08.032953 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.031663 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:31:08.032953 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.031940 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:31:08.090580 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.090550 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-istio-data\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:08.090802 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.090772 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dr77\" (UniqueName: \"kubernetes.io/projected/53bbece8-96fa-412b-b259-fece0b49a345-kube-api-access-5dr77\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:08.091001 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.090851 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-workload-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:08.091066 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.091043 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53bbece8-96fa-412b-b259-fece0b49a345-istio-podinfo\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:08.091121 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.091110 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-workload-certs\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:08.091160 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.091128 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53bbece8-96fa-412b-b259-fece0b49a345-istiod-ca-cert\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:08.091194 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.091167 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-istio-envoy\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:08.091257 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.091247 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-credential-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:08.091292 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.091268 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53bbece8-96fa-412b-b259-fece0b49a345-istio-token\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:08.192542 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.192455 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-credential-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:08.192542 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.192496 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53bbece8-96fa-412b-b259-fece0b49a345-istio-token\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:08.192765 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.192576 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-istio-data\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:08.192765 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.192635 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dr77\" (UniqueName: \"kubernetes.io/projected/53bbece8-96fa-412b-b259-fece0b49a345-kube-api-access-5dr77\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:08.192765 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.192685 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-workload-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:08.192765 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.192741 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53bbece8-96fa-412b-b259-fece0b49a345-istio-podinfo\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:08.193015 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.192816 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-workload-certs\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:08.193015 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.192843 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53bbece8-96fa-412b-b259-fece0b49a345-istiod-ca-cert\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:08.193015 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.192943 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-istio-envoy\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:08.193240 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.193217 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-credential-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:08.193548 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.193522 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-istio-data\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:08.193927 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.193885 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-workload-certs\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:08.194603 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.194578 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53bbece8-96fa-412b-b259-fece0b49a345-istiod-ca-cert\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:08.194912 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.194848 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-workload-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:08.198221 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.198197 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53bbece8-96fa-412b-b259-fece0b49a345-istio-podinfo\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:08.205757 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.205736 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-istio-envoy\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:08.216035 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.216013 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dr77\" (UniqueName: \"kubernetes.io/projected/53bbece8-96fa-412b-b259-fece0b49a345-kube-api-access-5dr77\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:08.216147 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.216131 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53bbece8-96fa-412b-b259-fece0b49a345-istio-token\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:08.349762 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.349730 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:08.679901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.679229 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc"] Oct 30 17:31:08.680333 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:31:08.679966 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53bbece8_96fa_412b_b259_fece0b49a345.slice/crio-9cc6d4c66ef839b1aa7ae3c9eb47ea4c8c92b4fd11b9552e39cd9bc94934901e WatchSource:0}: Error finding container 9cc6d4c66ef839b1aa7ae3c9eb47ea4c8c92b4fd11b9552e39cd9bc94934901e: Status 404 returned error can't find the container with id 9cc6d4c66ef839b1aa7ae3c9eb47ea4c8c92b4fd11b9552e39cd9bc94934901e Oct 30 17:31:08.682797 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.682766 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:31:08.682873 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.682827 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:31:08.682919 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:08.682872 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:31:09.642244 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:09.642211 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" event={"ID":"53bbece8-96fa-412b-b259-fece0b49a345","Type":"ContainerStarted","Data":"5b656994fd096c028872c0c2449bdedaee006c49528e97e39dc2fa758ae89a65"} Oct 30 17:31:09.642427 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:09.642263 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" event={"ID":"53bbece8-96fa-412b-b259-fece0b49a345","Type":"ContainerStarted","Data":"9cc6d4c66ef839b1aa7ae3c9eb47ea4c8c92b4fd11b9552e39cd9bc94934901e"} Oct 30 17:31:09.727652 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:09.727595 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" podStartSLOduration=2.727580006 podStartE2EDuration="2.727580006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:31:09.723981285 +0000 UTC m=+1127.776081257" watchObservedRunningTime="2026-10-30 17:31:09.727580006 +0000 UTC m=+1127.779679937" Oct 30 17:31:10.350223 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:10.350191 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:10.351783 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:10.351758 4965 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 17:31:10.351929 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:10.351803 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" podUID="53bbece8-96fa-412b-b259-fece0b49a345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 17:31:11.350192 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:11.350147 4965 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 17:31:11.350586 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:11.350209 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" podUID="53bbece8-96fa-412b-b259-fece0b49a345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 17:31:12.350338 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:12.350308 4965 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 17:31:12.350715 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:12.350358 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" podUID="53bbece8-96fa-412b-b259-fece0b49a345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 17:31:13.350018 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:13.349984 4965 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 17:31:13.350201 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:13.350048 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" podUID="53bbece8-96fa-412b-b259-fece0b49a345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 17:31:14.350658 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:14.350623 4965 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 17:31:14.351187 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:14.350683 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" podUID="53bbece8-96fa-412b-b259-fece0b49a345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 17:31:15.350803 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:15.350762 4965 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 17:31:15.351218 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:15.350832 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" podUID="53bbece8-96fa-412b-b259-fece0b49a345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 17:31:17.191630 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:17.191582 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:17.318258 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:17.318224 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:17.320526 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:17.320503 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:17.326230 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:17.326201 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:31:17.465246 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:17.465148 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:17.731936 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:17.731724 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:18.003944 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:18.001423 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:18.018968 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:18.018891 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:18.042508 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:18.042472 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:18.086914 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:18.048050 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:18.086914 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:18.049929 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 17:31:18.229912 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:18.229833 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 17:31:18.348215 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:18.348182 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:31:18.426050 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:18.426015 4965 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:31:18.426380 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:18.426325 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" podUID="5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:18.441385 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:18.441352 4965 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:31:18.441614 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:18.441586 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" podUID="d493c105-7ba4-4626-b438-2224db37a905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:18.974750 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:18.782950 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:18.974750 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:18.784044 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:31:18.998030 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:18.995534 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:31:19.415792 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:19.268837 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:19.415792 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:19.271684 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:31:19.547519 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:19.547490 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:31:19.557036 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:19.557007 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:31:23.229726 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.229693 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4"] Oct 30 17:31:23.235500 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.235481 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:23.257090 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.257058 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4"] Oct 30 17:31:23.340128 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.340096 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-istio-data\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:23.340321 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.340158 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4953033-0273-4681-8d46-94128ab21e6c-istio-podinfo\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:23.340321 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.340273 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jwq4\" (UniqueName: \"kubernetes.io/projected/f4953033-0273-4681-8d46-94128ab21e6c-kube-api-access-5jwq4\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:23.340321 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.340314 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-istio-envoy\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:23.340502 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.340480 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-credential-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:23.340621 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.340603 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-workload-certs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:23.340724 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.340706 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-workload-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:23.340764 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.340747 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4953033-0273-4681-8d46-94128ab21e6c-istiod-ca-cert\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:23.340819 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.340804 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4953033-0273-4681-8d46-94128ab21e6c-istio-token\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:23.442648 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.442609 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jwq4\" (UniqueName: \"kubernetes.io/projected/f4953033-0273-4681-8d46-94128ab21e6c-kube-api-access-5jwq4\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:23.442648 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.442653 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-istio-envoy\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:23.442941 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.442738 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-credential-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:23.442941 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.442850 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-workload-certs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:23.442941 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.442895 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-workload-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:23.442941 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.442920 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4953033-0273-4681-8d46-94128ab21e6c-istiod-ca-cert\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:23.443156 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.442946 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4953033-0273-4681-8d46-94128ab21e6c-istio-token\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:23.443156 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.443097 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-istio-data\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:23.443156 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.443146 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4953033-0273-4681-8d46-94128ab21e6c-istio-podinfo\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:23.443307 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.443284 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-workload-certs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:23.443360 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.443302 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-credential-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:23.443478 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.443449 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-istio-data\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:23.443629 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.443611 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4953033-0273-4681-8d46-94128ab21e6c-istiod-ca-cert\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:23.443674 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.443651 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-workload-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:23.445549 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.445524 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-istio-envoy\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:23.445969 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.445948 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4953033-0273-4681-8d46-94128ab21e6c-istio-podinfo\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:23.465598 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.465573 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jwq4\" (UniqueName: \"kubernetes.io/projected/f4953033-0273-4681-8d46-94128ab21e6c-kube-api-access-5jwq4\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:23.465822 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.465805 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4953033-0273-4681-8d46-94128ab21e6c-istio-token\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:23.551908 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.551812 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:23.879438 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.879407 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4"] Oct 30 17:31:23.880704 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:31:23.880673 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4953033_0273_4681_8d46_94128ab21e6c.slice/crio-13c888bef9e5f669f9fdcd727b9a06861ef0930a384c5c6679b76e3de29fc313 WatchSource:0}: Error finding container 13c888bef9e5f669f9fdcd727b9a06861ef0930a384c5c6679b76e3de29fc313: Status 404 returned error can't find the container with id 13c888bef9e5f669f9fdcd727b9a06861ef0930a384c5c6679b76e3de29fc313 Oct 30 17:31:23.883417 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.883382 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:31:23.883509 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.883448 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:31:23.883509 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:23.883475 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:31:24.521103 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:24.521049 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" event={"ID":"f4953033-0273-4681-8d46-94128ab21e6c","Type":"ContainerStarted","Data":"aa9c503d179ee92a95cc9bc385e86b667d072ce5f0deb021da3cbf0531623bc2"} Oct 30 17:31:24.521517 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:24.521130 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" event={"ID":"f4953033-0273-4681-8d46-94128ab21e6c","Type":"ContainerStarted","Data":"13c888bef9e5f669f9fdcd727b9a06861ef0930a384c5c6679b76e3de29fc313"} Oct 30 17:31:24.552153 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:24.552121 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:24.553710 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:24.553680 4965 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 17:31:24.553841 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:24.553722 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" podUID="f4953033-0273-4681-8d46-94128ab21e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 17:31:24.592197 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:24.592138 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" podStartSLOduration=1.59212269 podStartE2EDuration="1.59212269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:31:24.590414557 +0000 UTC m=+1142.642514487" watchObservedRunningTime="2026-10-30 17:31:24.59212269 +0000 UTC m=+1142.644222622" Oct 30 17:31:25.552921 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:25.552886 4965 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 17:31:25.553318 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:25.552936 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" podUID="f4953033-0273-4681-8d46-94128ab21e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 17:31:26.553027 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:26.552989 4965 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 17:31:26.553432 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:26.553043 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" podUID="f4953033-0273-4681-8d46-94128ab21e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 17:31:27.552169 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:27.552133 4965 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 17:31:27.552388 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:27.552193 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" podUID="f4953033-0273-4681-8d46-94128ab21e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 17:31:28.552521 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:28.552481 4965 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 17:31:28.552971 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:28.552543 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" podUID="f4953033-0273-4681-8d46-94128ab21e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 17:31:29.552213 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:29.552172 4965 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 17:31:29.552388 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:29.552240 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" podUID="f4953033-0273-4681-8d46-94128ab21e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 17:31:30.552551 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:30.552511 4965 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 17:31:30.552970 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:30.552588 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" podUID="f4953033-0273-4681-8d46-94128ab21e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 17:31:30.826628 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:30.826597 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:31:31.567642 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:31.567601 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:31.569696 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:31.569675 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:31.571183 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:31.571165 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:31:33.550996 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:33.550945 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:33.999091 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:33.979831 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:33.999373 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:33.995961 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:34.019985 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:34.019935 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:34.048998 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:34.048949 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:34.050065 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:34.050034 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:34.552031 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:34.551984 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 17:31:34.553379 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:34.553351 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:31:34.573002 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:34.572954 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 17:31:35.070712 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:35.070653 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:35.098607 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:35.098559 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:31:35.161926 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:35.154111 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 17:31:35.173891 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:35.171438 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:31:35.174206 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:35.174153 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:35.261568 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:35.181765 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 17:31:35.261568 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:35.181828 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 17:31:35.261568 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:35.250006 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:31:35.261568 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:35.250080 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:31:35.261568 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:35.181875 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 17:31:35.261568 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:35.252232 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:31:35.333254 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:35.281235 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:35.569509 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:35.567589 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:35.661119 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:35.661013 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 17:31:35.684939 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:35.673380 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:31:36.123022 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:36.122969 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:36.223017 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:36.191110 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 17:31:36.226298 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:36.226170 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:36.226834 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:36.226809 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:36.257737 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:36.257694 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:36.660724 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:36.646119 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 17:31:36.660724 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:36.648903 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:31:36.703928 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:31:36.661835 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:31:36.769414 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:36.760637 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:37.155884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:37.155321 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:31:37.273405 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:37.273359 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:31:37.278963 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:37.246688 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 17:31:37.317059 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:37.317029 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:31:37.330197 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:37.327518 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:37.379514 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:37.379472 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:31:37.396376 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:37.396353 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:31:37.396671 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:37.396655 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:31:54.241226 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:54.241095 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 17:31:54.243302 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:54.242003 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:54.352023 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:54.351978 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:54.412974 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:54.412932 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 17:31:54.631004 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:54.630957 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:54.697630 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:54.697582 4965 generic.go:356] "Generic (PLEG): container finished" podID="65150c77-1df1-42d6-b0f5-21ba7e248b59" containerID="069a0e9f3394469d0f518189a118fd8e70c721f83b54fa9b8fda18d67474874a" exitCode=0 Oct 30 17:31:54.698042 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:54.698001 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-phxks-zhmxv" event={"ID":"65150c77-1df1-42d6-b0f5-21ba7e248b59","Type":"ContainerDied","Data":"069a0e9f3394469d0f518189a118fd8e70c721f83b54fa9b8fda18d67474874a"} Oct 30 17:31:55.046015 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:55.041987 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 17:31:55.153184 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:55.153132 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:55.253499 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:55.253456 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 17:31:55.255244 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:55.255204 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:55.303612 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:55.274667 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:31:55.322043 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:55.322009 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:31:55.323519 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:55.275133 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:55.373937 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:55.373103 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:55.373937 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:55.373903 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:31:55.486005 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:55.485953 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:55.531890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:55.499453 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:31:55.994744 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:55.994709 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:31:56.011229 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:56.011198 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:31:56.056535 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:56.056500 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:31:56.056797 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:56.056781 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:31:56.380933 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:56.380904 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-phxks-zhmxv" Oct 30 17:31:56.563026 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:56.562988 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65150c77-1df1-42d6-b0f5-21ba7e248b59-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65150c77-1df1-42d6-b0f5-21ba7e248b59-trusted-ca-bundle\") pod \"65150c77-1df1-42d6-b0f5-21ba7e248b59\" (UID: \"65150c77-1df1-42d6-b0f5-21ba7e248b59\") " Oct 30 17:31:56.563224 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:56.563091 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65150c77-1df1-42d6-b0f5-21ba7e248b59-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/65150c77-1df1-42d6-b0f5-21ba7e248b59-archives-path\") pod \"65150c77-1df1-42d6-b0f5-21ba7e248b59\" (UID: \"65150c77-1df1-42d6-b0f5-21ba7e248b59\") " Oct 30 17:31:56.563224 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:56.563162 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65150c77-1df1-42d6-b0f5-21ba7e248b59-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65150c77-1df1-42d6-b0f5-21ba7e248b59-service-ca-bundle\") pod \"65150c77-1df1-42d6-b0f5-21ba7e248b59\" (UID: \"65150c77-1df1-42d6-b0f5-21ba7e248b59\") " Oct 30 17:31:56.563354 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:56.563285 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65150c77-1df1-42d6-b0f5-21ba7e248b59-kube-api-access-57cz9\" (UniqueName: \"kubernetes.io/projected/65150c77-1df1-42d6-b0f5-21ba7e248b59-kube-api-access-57cz9\") pod \"65150c77-1df1-42d6-b0f5-21ba7e248b59\" (UID: \"65150c77-1df1-42d6-b0f5-21ba7e248b59\") " Oct 30 17:31:56.564690 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:56.564662 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65150c77-1df1-42d6-b0f5-21ba7e248b59-archives-path" pod "65150c77-1df1-42d6-b0f5-21ba7e248b59" (UID: "65150c77-1df1-42d6-b0f5-21ba7e248b59"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:31:56.564999 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:56.564976 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65150c77-1df1-42d6-b0f5-21ba7e248b59-trusted-ca-bundle" pod "65150c77-1df1-42d6-b0f5-21ba7e248b59" (UID: "65150c77-1df1-42d6-b0f5-21ba7e248b59"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:31:56.565476 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:56.565445 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65150c77-1df1-42d6-b0f5-21ba7e248b59-service-ca-bundle" pod "65150c77-1df1-42d6-b0f5-21ba7e248b59" (UID: "65150c77-1df1-42d6-b0f5-21ba7e248b59"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:31:56.565571 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:56.565529 4965 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65150c77-1df1-42d6-b0f5-21ba7e248b59-trusted-ca-bundle\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:31:56.565571 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:56.565556 4965 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/65150c77-1df1-42d6-b0f5-21ba7e248b59-archives-path\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:31:56.567885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:56.567847 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65150c77-1df1-42d6-b0f5-21ba7e248b59-kube-api-access-57cz9" pod "65150c77-1df1-42d6-b0f5-21ba7e248b59" (UID: "65150c77-1df1-42d6-b0f5-21ba7e248b59"). InnerVolumeSpecName "kube-api-access-57cz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:31:56.667400 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:56.667325 4965 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65150c77-1df1-42d6-b0f5-21ba7e248b59-service-ca-bundle\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:31:56.667400 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:56.667354 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57cz9\" (UniqueName: \"kubernetes.io/projected/65150c77-1df1-42d6-b0f5-21ba7e248b59-kube-api-access-57cz9\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:31:57.067371 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:57.067296 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-phxks-zhmxv" Oct 30 17:31:57.067511 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:57.067289 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-phxks-zhmxv" event={"ID":"65150c77-1df1-42d6-b0f5-21ba7e248b59","Type":"ContainerDied","Data":"26faf166ecfcc2ea1f83c0dd06955f2463f14fcdc70b38d22a3aa976d449eefe"} Oct 30 17:31:57.067511 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:31:57.067415 4965 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26faf166ecfcc2ea1f83c0dd06955f2463f14fcdc70b38d22a3aa976d449eefe" Oct 30 17:32:05.997909 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:32:05.997854 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:32:06.526354 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:06.526320 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:32:22.253259 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:22.253227 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 17:32:22.255830 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:22.254789 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 17:32:31.021889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:31.021050 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 17:32:31.105897 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:31.105197 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:31.277266 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:31.275965 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:35.997032 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:32:35.996953 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:32:46.189432 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:46.189397 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:32:49.018887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:49.012958 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:49.275304 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:49.274082 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:49.601062 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:49.600994 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 17:32:49.766014 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:49.765954 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:50.064094 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:50.063983 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 17:32:50.179888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:50.049959 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:50.332890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:50.331177 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 17:32:50.332890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:50.331890 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 17:32:50.609945 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:50.602021 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded" Oct 30 17:32:50.707663 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:50.707615 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 17:32:50.717890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:50.709439 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 17:32:50.843886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:50.840000 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:51.036944 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:51.032758 4965 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:32:51.072888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:51.032851 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" podUID="ddb82e7c-5038-46b7-937e-c9ae4b12b2c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:51.126228 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:51.126163 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:51.143358 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:51.136923 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:51.364334 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:51.345197 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 17:32:51.364334 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:51.345875 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:51.402922 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:32:51.393048 4965 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.269s" Oct 30 17:32:51.402922 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:51.393136 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:32:51.402922 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:51.393184 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:32:51.402922 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:51.393269 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:32:51.402922 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:51.393300 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:32:51.402922 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:51.393332 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:32:51.402922 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:51.393362 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:32:51.402922 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:51.393393 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:32:51.431894 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:51.431851 4965 trace.go:236] Trace[73810587]: "Calculate volume metrics of manifests for pod opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg" (30-Oct-2026 17:32:49.914) (total time: 1517ms): Oct 30 17:32:51.431894 ip-10-0-139-222 kubenswrapper[4965]: Trace[73810587]: [1.517554151s] [1.517554151s] END Oct 30 17:32:51.834294 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:51.834250 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:51.840279 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:51.834549 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:51.898886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:51.866248 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:52.127221 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:52.127124 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:52.141381 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:52.141333 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:52.141756 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:52.141722 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:52.275885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:52.271658 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:32:52.358057 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:52.357991 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:52.358468 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:52.358432 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 17:32:52.850887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:52.842274 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:52.911942 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:52.889512 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:53.148945 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:53.148766 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:53.211889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:53.209294 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:32:53.285627 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:53.261304 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:32:53.285627 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:53.261352 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:32:53.285627 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:53.228593 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:53.310006 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:53.309978 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:32:53.312389 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:53.312354 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:32:53.325067 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:32:53.324609 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:33:04.394883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:04.393926 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:04.416587 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:04.416544 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:05.995807 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:33:05.995768 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:33:08.669366 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.668750 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn"] Oct 30 17:33:08.672027 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.670419 4965 cpu_manager.go:414] "containerMap: removing container" podUID="65150c77-1df1-42d6-b0f5-21ba7e248b59" containerName="insights-gathering" Oct 30 17:33:08.672027 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.670441 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65150c77-1df1-42d6-b0f5-21ba7e248b59" containerName="insights-gathering" Oct 30 17:33:08.672027 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.670792 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="65150c77-1df1-42d6-b0f5-21ba7e248b59" containerName="insights-gathering" Oct 30 17:33:08.676205 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.676180 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:08.700740 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.700713 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn"] Oct 30 17:33:08.717676 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.717645 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istio-podinfo\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:08.717834 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.717696 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istio-data\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:08.717834 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.717793 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-workload-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:08.717953 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.717906 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-workload-certs\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:08.717994 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.717984 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istio-envoy\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:08.718116 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.718090 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istiod-ca-cert\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:08.718208 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.718191 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2qx9\" (UniqueName: \"kubernetes.io/projected/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-kube-api-access-b2qx9\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:08.718301 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.718276 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-credential-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:08.718393 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.718376 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istio-token\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:08.806679 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.806649 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p"] Oct 30 17:33:08.810975 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.810955 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:08.819572 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.819534 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istio-podinfo\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:08.819697 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.819604 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istio-token\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:08.819697 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.819672 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istio-podinfo\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:08.819805 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.819706 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istio-data\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:08.819805 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.819739 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-workload-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:08.819805 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.819767 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istio-envoy\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:08.819982 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.819813 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-credential-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:08.819982 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.819846 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istio-token\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:08.819982 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.819891 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gb9n\" (UniqueName: \"kubernetes.io/projected/5e25ba26-3a38-490c-bc1a-199cfdc132a9-kube-api-access-4gb9n\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:08.819982 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.819921 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-workload-certs\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:08.819982 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.819960 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istiod-ca-cert\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:08.820295 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.820000 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istio-data\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:08.820295 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.820065 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istio-envoy\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:08.820295 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.820110 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-workload-certs\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:08.820295 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.820141 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istiod-ca-cert\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:08.820295 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.820190 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-workload-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:08.820295 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.820236 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2qx9\" (UniqueName: \"kubernetes.io/projected/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-kube-api-access-b2qx9\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:08.820557 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.820360 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-credential-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:08.820855 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.820836 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-credential-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:08.821627 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.821602 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-workload-certs\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:08.822094 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.822050 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istio-data\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:08.822345 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.822323 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istiod-ca-cert\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:08.822345 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.822332 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-workload-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:08.823599 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.823576 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istio-envoy\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:08.823738 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.823719 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istio-podinfo\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:08.833099 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.833059 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p"] Oct 30 17:33:08.845400 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.845375 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istio-token\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:08.845649 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.845631 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2qx9\" (UniqueName: \"kubernetes.io/projected/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-kube-api-access-b2qx9\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:08.921590 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.921509 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-workload-certs\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:08.921590 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.921562 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-workload-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:08.921822 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.921690 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istio-podinfo\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:08.921822 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.921797 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istio-envoy\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:08.921974 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.921842 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-credential-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:08.921974 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.921913 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istio-token\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:08.922144 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.922006 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-workload-certs\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:08.922144 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.922076 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-workload-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:08.922144 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.922102 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gb9n\" (UniqueName: \"kubernetes.io/projected/5e25ba26-3a38-490c-bc1a-199cfdc132a9-kube-api-access-4gb9n\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:08.922303 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.922172 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istiod-ca-cert\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:08.922303 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.922210 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istio-data\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:08.922420 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.922330 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-credential-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:08.922574 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.922554 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istio-data\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:08.922840 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.922818 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istiod-ca-cert\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:08.924291 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.924271 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istio-envoy\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:08.924446 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.924430 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istio-podinfo\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:08.944906 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.944884 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gb9n\" (UniqueName: \"kubernetes.io/projected/5e25ba26-3a38-490c-bc1a-199cfdc132a9-kube-api-access-4gb9n\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:08.945015 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.944955 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istio-token\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:08.990880 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:08.990846 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:09.128546 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:09.128513 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:09.333276 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:33:09.333241 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc2fac0f_42be_41e6_8288_cdaaeaab7a1d.slice/crio-5f804571e3363e57eda233754d979cbf65f3d00ece531bab16f32d0a830db0ee WatchSource:0}: Error finding container 5f804571e3363e57eda233754d979cbf65f3d00ece531bab16f32d0a830db0ee: Status 404 returned error can't find the container with id 5f804571e3363e57eda233754d979cbf65f3d00ece531bab16f32d0a830db0ee Oct 30 17:33:09.334129 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:09.334102 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn"] Oct 30 17:33:09.335990 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:09.335949 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:33:09.336094 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:09.336028 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:33:09.336094 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:09.336070 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:33:09.442106 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:09.442031 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p"] Oct 30 17:33:09.446322 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:33:09.446292 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e25ba26_3a38_490c_bc1a_199cfdc132a9.slice/crio-2c69aea38e4be741b7b71379d5bf4ad8d026e0302ab5e7cbf008a3d056216509 WatchSource:0}: Error finding container 2c69aea38e4be741b7b71379d5bf4ad8d026e0302ab5e7cbf008a3d056216509: Status 404 returned error can't find the container with id 2c69aea38e4be741b7b71379d5bf4ad8d026e0302ab5e7cbf008a3d056216509 Oct 30 17:33:09.449182 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:09.449145 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:33:09.449260 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:09.449217 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:33:09.449260 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:09.449248 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:33:09.904362 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:09.904322 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f8ac7b58-5dc5d74fc4-kskcv"] Oct 30 17:33:09.908697 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:09.908674 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f8ac7b58-5dc5d74fc4-kskcv" Oct 30 17:33:09.924699 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:09.924674 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f8ac7b58-dockercfg-2mqtb\"" Oct 30 17:33:09.928264 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:09.928241 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f8ac7b58-5dc5d74fc4-kskcv"] Oct 30 17:33:09.928989 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:09.928967 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f8ac7b58\"" Oct 30 17:33:09.951540 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:09.951505 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8239abfe-bed3-4eac-8387-a7eec36f9d8b-plugins-config-volume\") pod \"payload-processing-e2e-ait-f8ac7b58-5dc5d74fc4-kskcv\" (UID: \"8239abfe-bed3-4eac-8387-a7eec36f9d8b\") " pod="openshift-ingress/payload-processing-e2e-ait-f8ac7b58-5dc5d74fc4-kskcv" Oct 30 17:33:09.951711 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:09.951582 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwhbs\" (UniqueName: \"kubernetes.io/projected/8239abfe-bed3-4eac-8387-a7eec36f9d8b-kube-api-access-wwhbs\") pod \"payload-processing-e2e-ait-f8ac7b58-5dc5d74fc4-kskcv\" (UID: \"8239abfe-bed3-4eac-8387-a7eec36f9d8b\") " pod="openshift-ingress/payload-processing-e2e-ait-f8ac7b58-5dc5d74fc4-kskcv" Oct 30 17:33:09.994918 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:09.994624 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f8ac7b58-5d5f6b56bc-mt8dc"] Oct 30 17:33:10.061885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:10.058099 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8239abfe-bed3-4eac-8387-a7eec36f9d8b-plugins-config-volume\") pod \"payload-processing-e2e-ait-f8ac7b58-5dc5d74fc4-kskcv\" (UID: \"8239abfe-bed3-4eac-8387-a7eec36f9d8b\") " pod="openshift-ingress/payload-processing-e2e-ait-f8ac7b58-5dc5d74fc4-kskcv" Oct 30 17:33:10.061885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:10.058224 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwhbs\" (UniqueName: \"kubernetes.io/projected/8239abfe-bed3-4eac-8387-a7eec36f9d8b-kube-api-access-wwhbs\") pod \"payload-processing-e2e-ait-f8ac7b58-5dc5d74fc4-kskcv\" (UID: \"8239abfe-bed3-4eac-8387-a7eec36f9d8b\") " pod="openshift-ingress/payload-processing-e2e-ait-f8ac7b58-5dc5d74fc4-kskcv" Oct 30 17:33:10.061885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:10.059659 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8239abfe-bed3-4eac-8387-a7eec36f9d8b-plugins-config-volume\") pod \"payload-processing-e2e-ait-f8ac7b58-5dc5d74fc4-kskcv\" (UID: \"8239abfe-bed3-4eac-8387-a7eec36f9d8b\") " pod="openshift-ingress/payload-processing-e2e-ait-f8ac7b58-5dc5d74fc4-kskcv" Oct 30 17:33:10.211040 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:10.200999 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f8ac7b58-5d5f6b56bc-mt8dc" Oct 30 17:33:10.407925 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:10.402805 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a20e9e32-4b37-4268-b2d9-3e23b1607c52-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f8ac7b58-5d5f6b56bc-mt8dc\" (UID: \"a20e9e32-4b37-4268-b2d9-3e23b1607c52\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f8ac7b58-5d5f6b56bc-mt8dc" Oct 30 17:33:10.407925 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:10.403011 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k2nf\" (UniqueName: \"kubernetes.io/projected/a20e9e32-4b37-4268-b2d9-3e23b1607c52-kube-api-access-6k2nf\") pod \"payload-pre-processing-e2e-ait-f8ac7b58-5d5f6b56bc-mt8dc\" (UID: \"a20e9e32-4b37-4268-b2d9-3e23b1607c52\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f8ac7b58-5d5f6b56bc-mt8dc" Oct 30 17:33:10.439434 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:10.408615 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwhbs\" (UniqueName: \"kubernetes.io/projected/8239abfe-bed3-4eac-8387-a7eec36f9d8b-kube-api-access-wwhbs\") pod \"payload-processing-e2e-ait-f8ac7b58-5dc5d74fc4-kskcv\" (UID: \"8239abfe-bed3-4eac-8387-a7eec36f9d8b\") " pod="openshift-ingress/payload-processing-e2e-ait-f8ac7b58-5dc5d74fc4-kskcv" Oct 30 17:33:10.516936 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:10.508252 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a20e9e32-4b37-4268-b2d9-3e23b1607c52-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f8ac7b58-5d5f6b56bc-mt8dc\" (UID: \"a20e9e32-4b37-4268-b2d9-3e23b1607c52\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f8ac7b58-5d5f6b56bc-mt8dc" Oct 30 17:33:10.516936 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:10.508386 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k2nf\" (UniqueName: \"kubernetes.io/projected/a20e9e32-4b37-4268-b2d9-3e23b1607c52-kube-api-access-6k2nf\") pod \"payload-pre-processing-e2e-ait-f8ac7b58-5d5f6b56bc-mt8dc\" (UID: \"a20e9e32-4b37-4268-b2d9-3e23b1607c52\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f8ac7b58-5d5f6b56bc-mt8dc" Oct 30 17:33:10.527594 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:10.527327 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a20e9e32-4b37-4268-b2d9-3e23b1607c52-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f8ac7b58-5d5f6b56bc-mt8dc\" (UID: \"a20e9e32-4b37-4268-b2d9-3e23b1607c52\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f8ac7b58-5d5f6b56bc-mt8dc" Oct 30 17:33:10.528666 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:10.528612 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f8ac7b58-5dc5d74fc4-kskcv" Oct 30 17:33:10.534460 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:10.534439 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k2nf\" (UniqueName: \"kubernetes.io/projected/a20e9e32-4b37-4268-b2d9-3e23b1607c52-kube-api-access-6k2nf\") pod \"payload-pre-processing-e2e-ait-f8ac7b58-5d5f6b56bc-mt8dc\" (UID: \"a20e9e32-4b37-4268-b2d9-3e23b1607c52\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f8ac7b58-5d5f6b56bc-mt8dc" Oct 30 17:33:10.538832 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:10.538809 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f8ac7b58-5d5f6b56bc-mt8dc"] Oct 30 17:33:10.538951 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:10.538932 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" event={"ID":"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d","Type":"ContainerStarted","Data":"ce9f8f640e293cde94e421cfa710e735b7d2b251a893e107bb6cab6f6d3f36d1"} Oct 30 17:33:10.539015 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:10.538954 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" event={"ID":"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d","Type":"ContainerStarted","Data":"5f804571e3363e57eda233754d979cbf65f3d00ece531bab16f32d0a830db0ee"} Oct 30 17:33:10.539703 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:10.539683 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" event={"ID":"5e25ba26-3a38-490c-bc1a-199cfdc132a9","Type":"ContainerStarted","Data":"2cc49f3d03304661d4f2f2ba290b890c835ef30e7ec4d107a5de327d435b1bc3"} Oct 30 17:33:10.539740 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:10.539718 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" event={"ID":"5e25ba26-3a38-490c-bc1a-199cfdc132a9","Type":"ContainerStarted","Data":"2c69aea38e4be741b7b71379d5bf4ad8d026e0302ab5e7cbf008a3d056216509"} Oct 30 17:33:10.643669 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:10.643613 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" podStartSLOduration=2.643591174 podStartE2EDuration="2.643591174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:10.637150853 +0000 UTC m=+1248.689250784" watchObservedRunningTime="2026-10-30 17:33:10.643591174 +0000 UTC m=+1248.695691107" Oct 30 17:33:10.645554 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:10.645513 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" podStartSLOduration=2.645500743 podStartE2EDuration="2.645500743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:10.592234937 +0000 UTC m=+1248.644334896" watchObservedRunningTime="2026-10-30 17:33:10.645500743 +0000 UTC m=+1248.697600673" Oct 30 17:33:10.832731 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:10.832699 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f8ac7b58-5d5f6b56bc-mt8dc" Oct 30 17:33:10.939348 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:10.939316 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f8ac7b58-5dc5d74fc4-kskcv"] Oct 30 17:33:10.992603 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:10.992573 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:11.011462 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:11.011425 4965 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 17:33:11.017018 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:11.016972 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" podUID="cc2fac0f-42be-41e6-8288-cdaaeaab7a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 17:33:11.173951 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:11.158376 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:11.196628 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:11.196593 4965 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 17:33:11.197183 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:11.196857 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" podUID="5e25ba26-3a38-490c-bc1a-199cfdc132a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 17:33:11.805887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:11.805393 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f8ac7b58-5dc5d74fc4-kskcv" event={"ID":"8239abfe-bed3-4eac-8387-a7eec36f9d8b","Type":"ContainerStarted","Data":"30d9777b4536e72366b14f6bbd972306bee55bcacf6a119e2996073f8d04e2eb"} Oct 30 17:33:11.810970 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:11.810931 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f8ac7b58-5d5f6b56bc-mt8dc"] Oct 30 17:33:11.991887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:11.991231 4965 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 17:33:11.991887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:11.991301 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" podUID="cc2fac0f-42be-41e6-8288-cdaaeaab7a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 17:33:12.210447 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:12.142042 4965 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 17:33:12.210447 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:12.142106 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" podUID="5e25ba26-3a38-490c-bc1a-199cfdc132a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 17:33:12.822473 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:12.822437 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f8ac7b58-5dc5d74fc4-kskcv" event={"ID":"8239abfe-bed3-4eac-8387-a7eec36f9d8b","Type":"ContainerStarted","Data":"d98719bdc6ee6642de99788db0ad6f9701a5d6f2bb2ccb8e93323b2e6f872dd0"} Oct 30 17:33:12.822650 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:12.822540 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f8ac7b58-5dc5d74fc4-kskcv" Oct 30 17:33:12.827792 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:12.824462 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f8ac7b58-5d5f6b56bc-mt8dc" event={"ID":"a20e9e32-4b37-4268-b2d9-3e23b1607c52","Type":"ContainerStarted","Data":"83f518eb85b6e5e42f81450a4957b585ab1cdf1371d625a34bb06d68da8809dc"} Oct 30 17:33:12.827792 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:12.824506 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f8ac7b58-5d5f6b56bc-mt8dc" event={"ID":"a20e9e32-4b37-4268-b2d9-3e23b1607c52","Type":"ContainerStarted","Data":"db3a8df94457f846f4d8acef90bca9025bb9c835a5fa8b41b4526317cc4aa0ab"} Oct 30 17:33:12.827792 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:12.825067 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f8ac7b58-5d5f6b56bc-mt8dc" Oct 30 17:33:12.949966 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:12.949914 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f8ac7b58-5dc5d74fc4-kskcv" podStartSLOduration=3.9499002880000003 podStartE2EDuration="3.949900288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:12.908446268 +0000 UTC m=+1250.960546196" watchObservedRunningTime="2026-10-30 17:33:12.949900288 +0000 UTC m=+1251.002000226" Oct 30 17:33:12.952108 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:12.952068 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f8ac7b58-5d5f6b56bc-mt8dc" podStartSLOduration=3.952057826 podStartE2EDuration="3.952057826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:12.947679241 +0000 UTC m=+1250.999779169" watchObservedRunningTime="2026-10-30 17:33:12.952057826 +0000 UTC m=+1251.004157759" Oct 30 17:33:13.007935 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:12.993241 4965 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 17:33:13.007935 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:12.993317 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" podUID="cc2fac0f-42be-41e6-8288-cdaaeaab7a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 17:33:13.130884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:13.129297 4965 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 17:33:13.130884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:13.129362 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" podUID="5e25ba26-3a38-490c-bc1a-199cfdc132a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 17:33:14.007203 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:14.007162 4965 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 17:33:14.007418 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:14.007221 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" podUID="cc2fac0f-42be-41e6-8288-cdaaeaab7a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 17:33:14.129367 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:14.129331 4965 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 17:33:14.129750 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:14.129383 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" podUID="5e25ba26-3a38-490c-bc1a-199cfdc132a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 17:33:14.993891 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:14.992186 4965 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 17:33:14.993891 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:14.992249 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" podUID="cc2fac0f-42be-41e6-8288-cdaaeaab7a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 17:33:15.129696 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:15.129659 4965 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 17:33:15.130105 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:15.129710 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" podUID="5e25ba26-3a38-490c-bc1a-199cfdc132a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 17:33:16.003012 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.002801 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2"] Oct 30 17:33:16.007569 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.007534 4965 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 17:33:16.007726 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.007591 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" podUID="cc2fac0f-42be-41e6-8288-cdaaeaab7a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 17:33:16.019804 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.019762 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2"] Oct 30 17:33:16.019984 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.019939 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:16.035337 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.035294 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-15a954ad-openshift-default-dockercfg-5jxrd\"" Oct 30 17:33:16.045066 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.045031 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-workload-certs\") pod \"e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:16.045204 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.045079 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52968197-826b-42aa-8627-6287ec75aeed-istio-podinfo\") pod \"e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:16.045204 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.045114 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52968197-826b-42aa-8627-6287ec75aeed-istio-token\") pod \"e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:16.045204 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.045196 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-workload-socket\") pod \"e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:16.045393 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.045310 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-istio-data\") pod \"e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:16.045448 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.045417 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-istio-envoy\") pod \"e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:16.045506 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.045477 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqv7q\" (UniqueName: \"kubernetes.io/projected/52968197-826b-42aa-8627-6287ec75aeed-kube-api-access-nqv7q\") pod \"e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:16.045569 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.045514 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52968197-826b-42aa-8627-6287ec75aeed-istiod-ca-cert\") pod \"e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:16.045569 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.045550 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-credential-socket\") pod \"e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:16.168887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.149621 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-workload-socket\") pod \"e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:16.168887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.149758 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-istio-data\") pod \"e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:16.168887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.149879 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-istio-envoy\") pod \"e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:16.168887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.149939 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqv7q\" (UniqueName: \"kubernetes.io/projected/52968197-826b-42aa-8627-6287ec75aeed-kube-api-access-nqv7q\") pod \"e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:16.168887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.149974 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52968197-826b-42aa-8627-6287ec75aeed-istiod-ca-cert\") pod \"e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:16.168887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.150007 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-credential-socket\") pod \"e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:16.168887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.150083 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-workload-certs\") pod \"e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:16.168887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.150111 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52968197-826b-42aa-8627-6287ec75aeed-istio-podinfo\") pod \"e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:16.168887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.150151 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52968197-826b-42aa-8627-6287ec75aeed-istio-token\") pod \"e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:16.168887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.151626 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-workload-socket\") pod \"e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:16.168887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.151929 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-istio-data\") pod \"e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:16.214886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.187011 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52968197-826b-42aa-8627-6287ec75aeed-istiod-ca-cert\") pod \"e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:16.214886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.187213 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-workload-certs\") pod \"e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:16.214886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.187565 4965 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 17:33:16.214886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.187611 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" podUID="5e25ba26-3a38-490c-bc1a-199cfdc132a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 17:33:16.251825 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.251789 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-istio-envoy\") pod \"e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:16.257793 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.257683 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-credential-socket\") pod \"e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:16.296212 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.296132 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqv7q\" (UniqueName: \"kubernetes.io/projected/52968197-826b-42aa-8627-6287ec75aeed-kube-api-access-nqv7q\") pod \"e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:16.323946 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.323912 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52968197-826b-42aa-8627-6287ec75aeed-istio-token\") pod \"e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:16.716195 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.656838 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-84af0117-687b6d54bc-mn5pk"] Oct 30 17:33:16.744926 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.743016 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52968197-826b-42aa-8627-6287ec75aeed-istio-podinfo\") pod \"e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:16.750946 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.747781 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-84af0117-687b6d54bc-mn5pk"] Oct 30 17:33:16.767563 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.756460 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-84af0117-845ccbf768-mfwg2"] Oct 30 17:33:16.767563 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.759663 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-84af0117-687b6d54bc-mn5pk" Oct 30 17:33:16.789886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.785128 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-84af0117\"" Oct 30 17:33:16.818886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.815313 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-84af0117-845ccbf768-mfwg2" Oct 30 17:33:16.844905 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.837315 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e615e20e-ad32-4c76-b1aa-4b447b019699-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-84af0117-687b6d54bc-mn5pk\" (UID: \"e615e20e-ad32-4c76-b1aa-4b447b019699\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-84af0117-687b6d54bc-mn5pk" Oct 30 17:33:16.844905 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.837582 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt9cg\" (UniqueName: \"kubernetes.io/projected/e615e20e-ad32-4c76-b1aa-4b447b019699-kube-api-access-gt9cg\") pod \"payload-pre-processing-e2e-migrate-84af0117-687b6d54bc-mn5pk\" (UID: \"e615e20e-ad32-4c76-b1aa-4b447b019699\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-84af0117-687b6d54bc-mn5pk" Oct 30 17:33:16.849891 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.845810 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-84af0117-dockercfg-vz8xw\"" Oct 30 17:33:16.852901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.850168 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-84af0117-845ccbf768-mfwg2"] Oct 30 17:33:16.946890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.943961 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:16.946890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.944838 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e615e20e-ad32-4c76-b1aa-4b447b019699-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-84af0117-687b6d54bc-mn5pk\" (UID: \"e615e20e-ad32-4c76-b1aa-4b447b019699\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-84af0117-687b6d54bc-mn5pk" Oct 30 17:33:16.946890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.944903 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzlv5\" (UniqueName: \"kubernetes.io/projected/a4197c2a-53e7-436e-85ab-fd1417088a29-kube-api-access-fzlv5\") pod \"payload-processing-e2e-migrate-84af0117-845ccbf768-mfwg2\" (UID: \"a4197c2a-53e7-436e-85ab-fd1417088a29\") " pod="openshift-ingress/payload-processing-e2e-migrate-84af0117-845ccbf768-mfwg2" Oct 30 17:33:16.946890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.944980 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4197c2a-53e7-436e-85ab-fd1417088a29-plugins-config-volume\") pod \"payload-processing-e2e-migrate-84af0117-845ccbf768-mfwg2\" (UID: \"a4197c2a-53e7-436e-85ab-fd1417088a29\") " pod="openshift-ingress/payload-processing-e2e-migrate-84af0117-845ccbf768-mfwg2" Oct 30 17:33:16.946890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.945197 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt9cg\" (UniqueName: \"kubernetes.io/projected/e615e20e-ad32-4c76-b1aa-4b447b019699-kube-api-access-gt9cg\") pod \"payload-pre-processing-e2e-migrate-84af0117-687b6d54bc-mn5pk\" (UID: \"e615e20e-ad32-4c76-b1aa-4b447b019699\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-84af0117-687b6d54bc-mn5pk" Oct 30 17:33:16.946890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:16.946743 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e615e20e-ad32-4c76-b1aa-4b447b019699-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-84af0117-687b6d54bc-mn5pk\" (UID: \"e615e20e-ad32-4c76-b1aa-4b447b019699\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-84af0117-687b6d54bc-mn5pk" Oct 30 17:33:17.015152 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:17.012177 4965 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 17:33:17.015152 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:17.012688 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" podUID="cc2fac0f-42be-41e6-8288-cdaaeaab7a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 17:33:17.047542 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:17.047510 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt9cg\" (UniqueName: \"kubernetes.io/projected/e615e20e-ad32-4c76-b1aa-4b447b019699-kube-api-access-gt9cg\") pod \"payload-pre-processing-e2e-migrate-84af0117-687b6d54bc-mn5pk\" (UID: \"e615e20e-ad32-4c76-b1aa-4b447b019699\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-84af0117-687b6d54bc-mn5pk" Oct 30 17:33:17.053774 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:17.053274 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4197c2a-53e7-436e-85ab-fd1417088a29-plugins-config-volume\") pod \"payload-processing-e2e-migrate-84af0117-845ccbf768-mfwg2\" (UID: \"a4197c2a-53e7-436e-85ab-fd1417088a29\") " pod="openshift-ingress/payload-processing-e2e-migrate-84af0117-845ccbf768-mfwg2" Oct 30 17:33:17.053774 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:17.053669 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzlv5\" (UniqueName: \"kubernetes.io/projected/a4197c2a-53e7-436e-85ab-fd1417088a29-kube-api-access-fzlv5\") pod \"payload-processing-e2e-migrate-84af0117-845ccbf768-mfwg2\" (UID: \"a4197c2a-53e7-436e-85ab-fd1417088a29\") " pod="openshift-ingress/payload-processing-e2e-migrate-84af0117-845ccbf768-mfwg2" Oct 30 17:33:17.055039 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:17.054930 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4197c2a-53e7-436e-85ab-fd1417088a29-plugins-config-volume\") pod \"payload-processing-e2e-migrate-84af0117-845ccbf768-mfwg2\" (UID: \"a4197c2a-53e7-436e-85ab-fd1417088a29\") " pod="openshift-ingress/payload-processing-e2e-migrate-84af0117-845ccbf768-mfwg2" Oct 30 17:33:17.105843 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:17.105780 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzlv5\" (UniqueName: \"kubernetes.io/projected/a4197c2a-53e7-436e-85ab-fd1417088a29-kube-api-access-fzlv5\") pod \"payload-processing-e2e-migrate-84af0117-845ccbf768-mfwg2\" (UID: \"a4197c2a-53e7-436e-85ab-fd1417088a29\") " pod="openshift-ingress/payload-processing-e2e-migrate-84af0117-845ccbf768-mfwg2" Oct 30 17:33:17.129725 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:17.129689 4965 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 17:33:17.129903 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:17.129763 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" podUID="5e25ba26-3a38-490c-bc1a-199cfdc132a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 17:33:17.362943 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:17.341361 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-84af0117-845ccbf768-mfwg2" Oct 30 17:33:17.461395 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:17.460090 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2"] Oct 30 17:33:17.533731 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:17.533463 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-84af0117-687b6d54bc-mn5pk" Oct 30 17:33:17.553260 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:17.553215 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:33:17.553373 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:17.553310 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:33:17.553373 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:17.553354 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:33:17.988825 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:17.988740 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-84af0117-687b6d54bc-mn5pk"] Oct 30 17:33:18.006094 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:18.005965 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" event={"ID":"52968197-826b-42aa-8627-6287ec75aeed","Type":"ContainerStarted","Data":"89db0fcb3c90155b50f97695c029fef9179d59b03667925d93f8c6608290b838"} Oct 30 17:33:18.006094 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:18.006051 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" event={"ID":"52968197-826b-42aa-8627-6287ec75aeed","Type":"ContainerStarted","Data":"518d74a834e222ad3641b0eefc9813b48e797267d28c38443b92449bdbb4a717"} Oct 30 17:33:18.007616 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:33:18.007558 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode615e20e_ad32_4c76_b1aa_4b447b019699.slice/crio-b72dfc86ac1974c3ff62d2f7b6502bc5e443c842e74643c94bb2cd2dc349f15b WatchSource:0}: Error finding container b72dfc86ac1974c3ff62d2f7b6502bc5e443c842e74643c94bb2cd2dc349f15b: Status 404 returned error can't find the container with id b72dfc86ac1974c3ff62d2f7b6502bc5e443c842e74643c94bb2cd2dc349f15b Oct 30 17:33:18.040215 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:18.040183 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-84af0117-845ccbf768-mfwg2"] Oct 30 17:33:18.196890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:18.196838 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:18.215109 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:18.204300 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:18.236903 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:18.234314 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:33:18.303887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:18.286257 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" podStartSLOduration=3.286233986 podStartE2EDuration="3.286233986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:18.05964562 +0000 UTC m=+1256.111745573" watchObservedRunningTime="2026-10-30 17:33:18.286233986 +0000 UTC m=+1256.338333918" Oct 30 17:33:18.700341 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:18.700309 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:18.700803 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:18.700781 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:18.701641 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:18.701620 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:33:18.946996 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:18.946962 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:18.953519 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:18.953455 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:19.013726 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:19.013693 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-84af0117-687b6d54bc-mn5pk" event={"ID":"e615e20e-ad32-4c76-b1aa-4b447b019699","Type":"ContainerStarted","Data":"23252fa4cebb2cef94a3c345367e9750346da7f71148561ad12d0e231348a3fd"} Oct 30 17:33:19.013932 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:19.013743 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-84af0117-687b6d54bc-mn5pk" event={"ID":"e615e20e-ad32-4c76-b1aa-4b447b019699","Type":"ContainerStarted","Data":"b72dfc86ac1974c3ff62d2f7b6502bc5e443c842e74643c94bb2cd2dc349f15b"} Oct 30 17:33:19.013932 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:19.013771 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-84af0117-687b6d54bc-mn5pk" Oct 30 17:33:19.015453 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:19.015426 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-84af0117-845ccbf768-mfwg2" event={"ID":"a4197c2a-53e7-436e-85ab-fd1417088a29","Type":"ContainerStarted","Data":"5015d4432315dcb769ed6870f1a9e0ef62b79f61beef2c456e62ce6535f98d5b"} Oct 30 17:33:19.015560 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:19.015475 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-84af0117-845ccbf768-mfwg2" event={"ID":"a4197c2a-53e7-436e-85ab-fd1417088a29","Type":"ContainerStarted","Data":"59e39a4bfe764e4e3b3cbac87b17b9778f1ccfbb9e89db131c9a2c013188ed78"} Oct 30 17:33:19.015948 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:19.015928 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-84af0117-845ccbf768-mfwg2" Oct 30 17:33:19.016052 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:19.016019 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:19.016444 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:19.016427 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:33:19.053853 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:19.053813 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-84af0117-687b6d54bc-mn5pk" podStartSLOduration=3.053801856 podStartE2EDuration="3.053801856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:19.052478844 +0000 UTC m=+1257.104578774" watchObservedRunningTime="2026-10-30 17:33:19.053801856 +0000 UTC m=+1257.105901786" Oct 30 17:33:19.139841 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:19.139776 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-84af0117-845ccbf768-mfwg2" podStartSLOduration=3.139758719 podStartE2EDuration="3.139758719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:19.134595479 +0000 UTC m=+1257.186695410" watchObservedRunningTime="2026-10-30 17:33:19.139758719 +0000 UTC m=+1257.191858655" Oct 30 17:33:19.387899 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:19.387844 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:33:22.881033 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:22.880986 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 17:33:23.187938 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:33:23.187497 4965 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.05s" Oct 30 17:33:23.187938 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:23.187538 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-15a954ad-6c5fb7c4c6-vht7j"] Oct 30 17:33:23.191892 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:23.189943 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 17:33:23.336884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:23.301122 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:23.370539 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:23.341038 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:23.457042 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:23.436638 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 17:33:23.618915 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:23.617967 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-15a954ad-6c5fb7c4c6-vht7j"] Oct 30 17:33:23.618915 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:23.618094 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-15a954ad-8b6c47b87-76tsj"] Oct 30 17:33:23.629427 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:23.624549 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-15a954ad-6c5fb7c4c6-vht7j" Oct 30 17:33:23.692894 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:23.666794 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-15a954ad-8b6c47b87-76tsj"] Oct 30 17:33:23.693187 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:23.670221 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-15a954ad-8b6c47b87-76tsj" Oct 30 17:33:23.809942 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:23.809321 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-15a954ad-dockercfg-blqmg\"" Oct 30 17:33:23.913178 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:23.913137 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded" Oct 30 17:33:23.934923 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:23.914098 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 17:33:23.942332 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:23.914141 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 17:33:23.943127 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:23.922228 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-15a954ad\"" Oct 30 17:33:23.962964 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:23.940468 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpnxp\" (UniqueName: \"kubernetes.io/projected/cf726f6a-62b3-4030-9860-58f0fedbf57c-kube-api-access-hpnxp\") pod \"payload-processing-e2e-derive-15a954ad-6c5fb7c4c6-vht7j\" (UID: \"cf726f6a-62b3-4030-9860-58f0fedbf57c\") " pod="openshift-ingress/payload-processing-e2e-derive-15a954ad-6c5fb7c4c6-vht7j" Oct 30 17:33:23.978933 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:23.963426 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf726f6a-62b3-4030-9860-58f0fedbf57c-plugins-config-volume\") pod \"payload-processing-e2e-derive-15a954ad-6c5fb7c4c6-vht7j\" (UID: \"cf726f6a-62b3-4030-9860-58f0fedbf57c\") " pod="openshift-ingress/payload-processing-e2e-derive-15a954ad-6c5fb7c4c6-vht7j" Oct 30 17:33:24.135985 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.064120 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5693cca8-8479-4cda-b509-2ee15ef9a3b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-15a954ad-8b6c47b87-76tsj\" (UID: \"5693cca8-8479-4cda-b509-2ee15ef9a3b1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-15a954ad-8b6c47b87-76tsj" Oct 30 17:33:24.135985 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.064268 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bktzv\" (UniqueName: \"kubernetes.io/projected/5693cca8-8479-4cda-b509-2ee15ef9a3b1-kube-api-access-bktzv\") pod \"payload-pre-processing-e2e-derive-15a954ad-8b6c47b87-76tsj\" (UID: \"5693cca8-8479-4cda-b509-2ee15ef9a3b1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-15a954ad-8b6c47b87-76tsj" Oct 30 17:33:24.135985 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:23.968945 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:24.135985 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:23.945579 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:33:24.135985 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.065263 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f8ac7b58-5dc5d74fc4-kskcv" Oct 30 17:33:24.233889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.180039 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5693cca8-8479-4cda-b509-2ee15ef9a3b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-15a954ad-8b6c47b87-76tsj\" (UID: \"5693cca8-8479-4cda-b509-2ee15ef9a3b1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-15a954ad-8b6c47b87-76tsj" Oct 30 17:33:24.233889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.180154 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bktzv\" (UniqueName: \"kubernetes.io/projected/5693cca8-8479-4cda-b509-2ee15ef9a3b1-kube-api-access-bktzv\") pod \"payload-pre-processing-e2e-derive-15a954ad-8b6c47b87-76tsj\" (UID: \"5693cca8-8479-4cda-b509-2ee15ef9a3b1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-15a954ad-8b6c47b87-76tsj" Oct 30 17:33:24.233889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.180260 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpnxp\" (UniqueName: \"kubernetes.io/projected/cf726f6a-62b3-4030-9860-58f0fedbf57c-kube-api-access-hpnxp\") pod \"payload-processing-e2e-derive-15a954ad-6c5fb7c4c6-vht7j\" (UID: \"cf726f6a-62b3-4030-9860-58f0fedbf57c\") " pod="openshift-ingress/payload-processing-e2e-derive-15a954ad-6c5fb7c4c6-vht7j" Oct 30 17:33:24.233889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.180402 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf726f6a-62b3-4030-9860-58f0fedbf57c-plugins-config-volume\") pod \"payload-processing-e2e-derive-15a954ad-6c5fb7c4c6-vht7j\" (UID: \"cf726f6a-62b3-4030-9860-58f0fedbf57c\") " pod="openshift-ingress/payload-processing-e2e-derive-15a954ad-6c5fb7c4c6-vht7j" Oct 30 17:33:24.233889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.181514 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf726f6a-62b3-4030-9860-58f0fedbf57c-plugins-config-volume\") pod \"payload-processing-e2e-derive-15a954ad-6c5fb7c4c6-vht7j\" (UID: \"cf726f6a-62b3-4030-9860-58f0fedbf57c\") " pod="openshift-ingress/payload-processing-e2e-derive-15a954ad-6c5fb7c4c6-vht7j" Oct 30 17:33:24.233889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.217065 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5693cca8-8479-4cda-b509-2ee15ef9a3b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-15a954ad-8b6c47b87-76tsj\" (UID: \"5693cca8-8479-4cda-b509-2ee15ef9a3b1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-15a954ad-8b6c47b87-76tsj" Oct 30 17:33:24.242489 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.242446 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:24.398940 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.398826 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 17:33:24.399384 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.399352 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:24.405624 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.405598 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7"] Oct 30 17:33:24.477891 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.471250 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:24.635883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.615724 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bktzv\" (UniqueName: \"kubernetes.io/projected/5693cca8-8479-4cda-b509-2ee15ef9a3b1-kube-api-access-bktzv\") pod \"payload-pre-processing-e2e-derive-15a954ad-8b6c47b87-76tsj\" (UID: \"5693cca8-8479-4cda-b509-2ee15ef9a3b1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-15a954ad-8b6c47b87-76tsj" Oct 30 17:33:24.771738 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.697429 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:33:24.804062 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.804027 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:33:24.804413 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.804392 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:33:24.804560 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.804546 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7"] Oct 30 17:33:24.804751 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.804737 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:33:24.804945 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.804931 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:33:24.824969 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.697513 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:24.826589 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.812370 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpnxp\" (UniqueName: \"kubernetes.io/projected/cf726f6a-62b3-4030-9860-58f0fedbf57c-kube-api-access-hpnxp\") pod \"payload-processing-e2e-derive-15a954ad-6c5fb7c4c6-vht7j\" (UID: \"cf726f6a-62b3-4030-9860-58f0fedbf57c\") " pod="openshift-ingress/payload-processing-e2e-derive-15a954ad-6c5fb7c4c6-vht7j" Oct 30 17:33:24.832503 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.832478 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:33:24.833784 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.833761 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:33:24.838850 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.837436 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-workload-certs\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:24.838850 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.837624 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-workload-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:24.838850 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.838494 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-15a954ad-6c5fb7c4c6-vht7j" Oct 30 17:33:24.839079 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.838944 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-credential-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:24.839385 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.839363 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:33:24.848498 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.848481 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f8ac7b58-5d5f6b56bc-mt8dc" Oct 30 17:33:24.921122 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.921095 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-15a954ad-8b6c47b87-76tsj" Oct 30 17:33:24.928252 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.927944 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:33:24.943960 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.943920 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-istio-data\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:24.944114 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.944030 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-workload-certs\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:24.944114 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.944102 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c3202ad-345e-4bb9-8987-c287fdef26a6-istio-podinfo\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:24.944225 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.944151 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtwlh\" (UniqueName: \"kubernetes.io/projected/9c3202ad-345e-4bb9-8987-c287fdef26a6-kube-api-access-vtwlh\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:24.944293 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.944229 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-istio-envoy\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:24.944344 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.944313 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c3202ad-345e-4bb9-8987-c287fdef26a6-istio-token\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:24.944400 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.944391 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-workload-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:24.944597 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.944579 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-credential-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:24.944680 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.944612 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c3202ad-345e-4bb9-8987-c287fdef26a6-istiod-ca-cert\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:24.946592 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.946563 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-workload-certs\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:24.948474 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.948445 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-workload-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:24.950479 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:24.950454 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-credential-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:25.048552 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:25.046745 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-istio-envoy\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:25.048552 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:25.046828 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c3202ad-345e-4bb9-8987-c287fdef26a6-istio-token\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:25.048552 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:25.046997 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c3202ad-345e-4bb9-8987-c287fdef26a6-istiod-ca-cert\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:25.048552 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:25.047105 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-istio-data\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:25.048552 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:25.047201 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c3202ad-345e-4bb9-8987-c287fdef26a6-istio-podinfo\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:25.048552 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:25.047236 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtwlh\" (UniqueName: \"kubernetes.io/projected/9c3202ad-345e-4bb9-8987-c287fdef26a6-kube-api-access-vtwlh\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:25.048552 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:25.048391 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-istio-data\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:25.048552 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:25.048496 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c3202ad-345e-4bb9-8987-c287fdef26a6-istiod-ca-cert\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:25.051139 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:25.051110 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-istio-envoy\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:25.052225 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:25.052184 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c3202ad-345e-4bb9-8987-c287fdef26a6-istio-podinfo\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:25.104209 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:25.104179 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c3202ad-345e-4bb9-8987-c287fdef26a6-istio-token\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:25.105800 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:25.105778 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtwlh\" (UniqueName: \"kubernetes.io/projected/9c3202ad-345e-4bb9-8987-c287fdef26a6-kube-api-access-vtwlh\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:25.259019 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:25.258983 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-15a954ad-6c5fb7c4c6-vht7j"] Oct 30 17:33:25.263423 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:33:25.263395 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf726f6a_62b3_4030_9860_58f0fedbf57c.slice/crio-ee0eb95b565c855bcd2e8b913dfb15503b8059693360563144772a133108f016 WatchSource:0}: Error finding container ee0eb95b565c855bcd2e8b913dfb15503b8059693360563144772a133108f016: Status 404 returned error can't find the container with id ee0eb95b565c855bcd2e8b913dfb15503b8059693360563144772a133108f016 Oct 30 17:33:25.268193 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:25.268170 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:25.296281 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:25.296257 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-15a954ad-8b6c47b87-76tsj"] Oct 30 17:33:25.618589 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:25.618523 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7"] Oct 30 17:33:25.619354 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:33:25.619327 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c3202ad_345e_4bb9_8987_c287fdef26a6.slice/crio-b960a6b13533c956dc3dbc4dbf18919925c39b1526462d6c643d2a5417446593 WatchSource:0}: Error finding container b960a6b13533c956dc3dbc4dbf18919925c39b1526462d6c643d2a5417446593: Status 404 returned error can't find the container with id b960a6b13533c956dc3dbc4dbf18919925c39b1526462d6c643d2a5417446593 Oct 30 17:33:25.622515 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:25.622478 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:33:25.622582 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:25.622546 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:33:25.622615 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:25.622580 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:33:25.855276 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:25.855235 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-15a954ad-8b6c47b87-76tsj" event={"ID":"5693cca8-8479-4cda-b509-2ee15ef9a3b1","Type":"ContainerStarted","Data":"cf2ce50e0db13122d3f10170320d1d783fc058b28c68afd4c250bb55f63f0b2d"} Oct 30 17:33:25.855519 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:25.855493 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-15a954ad-8b6c47b87-76tsj" event={"ID":"5693cca8-8479-4cda-b509-2ee15ef9a3b1","Type":"ContainerStarted","Data":"9bb26f5548ce50fa919e628e4ba2551f4be095d506dc020df2600dc062dd8de6"} Oct 30 17:33:25.855698 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:25.855667 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-15a954ad-8b6c47b87-76tsj" Oct 30 17:33:25.857616 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:25.857592 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" event={"ID":"9c3202ad-345e-4bb9-8987-c287fdef26a6","Type":"ContainerStarted","Data":"a6d4c776a2d0b8587e28290d0860da6c84d7754a1ab82136e38ce598da91ade9"} Oct 30 17:33:25.857735 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:25.857633 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" event={"ID":"9c3202ad-345e-4bb9-8987-c287fdef26a6","Type":"ContainerStarted","Data":"b960a6b13533c956dc3dbc4dbf18919925c39b1526462d6c643d2a5417446593"} Oct 30 17:33:25.860322 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:25.860293 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-15a954ad-6c5fb7c4c6-vht7j" event={"ID":"cf726f6a-62b3-4030-9860-58f0fedbf57c","Type":"ContainerStarted","Data":"302359dea91cacc8001f6c0b6cf8af6bc2a7fd474e77c9f98521ce14c4f28ad7"} Oct 30 17:33:25.860442 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:25.860333 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-15a954ad-6c5fb7c4c6-vht7j" event={"ID":"cf726f6a-62b3-4030-9860-58f0fedbf57c","Type":"ContainerStarted","Data":"ee0eb95b565c855bcd2e8b913dfb15503b8059693360563144772a133108f016"} Oct 30 17:33:25.860442 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:25.860353 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-15a954ad-6c5fb7c4c6-vht7j" Oct 30 17:33:25.898784 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:25.898670 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-15a954ad-8b6c47b87-76tsj" podStartSLOduration=3.898656168 podStartE2EDuration="3.898656168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:25.896203506 +0000 UTC m=+1263.948303456" watchObservedRunningTime="2026-10-30 17:33:25.898656168 +0000 UTC m=+1263.950756098" Oct 30 17:33:25.952387 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:25.952326 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" podStartSLOduration=2.952306101 podStartE2EDuration="2.952306101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:25.946818286 +0000 UTC m=+1263.998918243" watchObservedRunningTime="2026-10-30 17:33:25.952306101 +0000 UTC m=+1264.004406036" Oct 30 17:33:25.990454 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:25.990399 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-15a954ad-6c5fb7c4c6-vht7j" podStartSLOduration=3.990386005 podStartE2EDuration="3.990386005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:25.989096305 +0000 UTC m=+1264.041196237" watchObservedRunningTime="2026-10-30 17:33:25.990386005 +0000 UTC m=+1264.042485936" Oct 30 17:33:26.270661 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:26.268415 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:26.270661 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:26.270518 4965 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 17:33:26.270661 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:26.270579 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" podUID="9c3202ad-345e-4bb9-8987-c287fdef26a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 17:33:27.270697 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:27.270255 4965 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 17:33:27.270697 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:27.270315 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" podUID="9c3202ad-345e-4bb9-8987-c287fdef26a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 17:33:28.280298 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:28.280256 4965 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 17:33:28.282900 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:28.280884 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" podUID="9c3202ad-345e-4bb9-8987-c287fdef26a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 17:33:29.280885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:29.280216 4965 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 17:33:29.280885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:29.280283 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" podUID="9c3202ad-345e-4bb9-8987-c287fdef26a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 17:33:30.024961 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:30.024934 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-84af0117-687b6d54bc-mn5pk" Oct 30 17:33:30.025133 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:30.025000 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-84af0117-845ccbf768-mfwg2" Oct 30 17:33:30.273909 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:30.271256 4965 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 17:33:30.273909 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:30.271316 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" podUID="9c3202ad-345e-4bb9-8987-c287fdef26a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 17:33:31.276889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:31.271310 4965 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 17:33:31.276889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:31.271386 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" podUID="9c3202ad-345e-4bb9-8987-c287fdef26a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 17:33:32.269570 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:32.269488 4965 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 17:33:32.269570 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:32.269553 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" podUID="9c3202ad-345e-4bb9-8987-c287fdef26a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 17:33:33.283402 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:33.277700 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:33.283402 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:33.279531 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:33.291885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:33.291474 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:33:36.001138 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:33:36.001105 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:33:36.894959 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:36.894933 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-15a954ad-8b6c47b87-76tsj" Oct 30 17:33:36.895139 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:36.895004 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-15a954ad-6c5fb7c4c6-vht7j" Oct 30 17:33:47.660887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:47.651850 4965 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:33:47.660887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:47.651939 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" podUID="d8f96554-6aec-4052-8219-307fb11b6404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:47.695889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:47.691233 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7"] Oct 30 17:33:47.699912 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:47.698570 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" Oct 30 17:33:47.717257 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:47.710708 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7"] Oct 30 17:33:47.719261 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:47.719230 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-306aed\"/\"openshift-service-ca.crt\"" Oct 30 17:33:47.719637 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:47.719617 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-306aed\"/\"default-dockercfg-4sfjv\"" Oct 30 17:33:47.731887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:47.719972 4965 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:33:47.731887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:47.720032 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" podUID="c7db6ef5-c274-4ca7-b20b-e2eb17592720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:47.731887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:47.720825 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-306aed\"/\"kube-root-ca.crt\"" Oct 30 17:33:47.732200 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:47.719853 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-306aed\"/\"distinct-w3-306aed-kserve-self-signed-certs\"" Oct 30 17:33:47.836760 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:47.821547 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d174a74f-7205-4020-94b0-e30652c9b721-model-cache\") pod \"distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7\" (UID: \"d174a74f-7205-4020-94b0-e30652c9b721\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" Oct 30 17:33:47.836760 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:47.821777 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d174a74f-7205-4020-94b0-e30652c9b721-tmp-dir\") pod \"distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7\" (UID: \"d174a74f-7205-4020-94b0-e30652c9b721\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" Oct 30 17:33:47.836760 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:47.821875 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d174a74f-7205-4020-94b0-e30652c9b721-home\") pod \"distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7\" (UID: \"d174a74f-7205-4020-94b0-e30652c9b721\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" Oct 30 17:33:47.836760 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:47.821943 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d174a74f-7205-4020-94b0-e30652c9b721-tls-certs\") pod \"distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7\" (UID: \"d174a74f-7205-4020-94b0-e30652c9b721\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" Oct 30 17:33:47.836760 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:47.822064 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4d7p\" (UniqueName: \"kubernetes.io/projected/d174a74f-7205-4020-94b0-e30652c9b721-kube-api-access-w4d7p\") pod \"distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7\" (UID: \"d174a74f-7205-4020-94b0-e30652c9b721\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" Oct 30 17:33:47.836760 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:47.822120 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d174a74f-7205-4020-94b0-e30652c9b721-dshm\") pod \"distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7\" (UID: \"d174a74f-7205-4020-94b0-e30652c9b721\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" Oct 30 17:33:47.836760 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:47.835970 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:47.935936 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:47.935003 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:47.972888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:47.970141 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 17:33:47.990888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:47.987814 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d174a74f-7205-4020-94b0-e30652c9b721-model-cache\") pod \"distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7\" (UID: \"d174a74f-7205-4020-94b0-e30652c9b721\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" Oct 30 17:33:47.990888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:47.987877 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d174a74f-7205-4020-94b0-e30652c9b721-tmp-dir\") pod \"distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7\" (UID: \"d174a74f-7205-4020-94b0-e30652c9b721\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" Oct 30 17:33:47.990888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:47.987931 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d174a74f-7205-4020-94b0-e30652c9b721-home\") pod \"distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7\" (UID: \"d174a74f-7205-4020-94b0-e30652c9b721\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" Oct 30 17:33:47.990888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:47.987982 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d174a74f-7205-4020-94b0-e30652c9b721-tls-certs\") pod \"distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7\" (UID: \"d174a74f-7205-4020-94b0-e30652c9b721\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" Oct 30 17:33:47.990888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:47.988082 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4d7p\" (UniqueName: \"kubernetes.io/projected/d174a74f-7205-4020-94b0-e30652c9b721-kube-api-access-w4d7p\") pod \"distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7\" (UID: \"d174a74f-7205-4020-94b0-e30652c9b721\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" Oct 30 17:33:47.990888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:47.988128 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d174a74f-7205-4020-94b0-e30652c9b721-dshm\") pod \"distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7\" (UID: \"d174a74f-7205-4020-94b0-e30652c9b721\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" Oct 30 17:33:48.040901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.030927 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d174a74f-7205-4020-94b0-e30652c9b721-home\") pod \"distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7\" (UID: \"d174a74f-7205-4020-94b0-e30652c9b721\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" Oct 30 17:33:48.040901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.031199 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d174a74f-7205-4020-94b0-e30652c9b721-tmp-dir\") pod \"distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7\" (UID: \"d174a74f-7205-4020-94b0-e30652c9b721\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" Oct 30 17:33:48.041632 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.041554 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d174a74f-7205-4020-94b0-e30652c9b721-model-cache\") pod \"distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7\" (UID: \"d174a74f-7205-4020-94b0-e30652c9b721\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" Oct 30 17:33:48.192946 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.192646 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 17:33:48.233227 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.193574 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 17:33:48.282945 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.281332 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4d7p\" (UniqueName: \"kubernetes.io/projected/d174a74f-7205-4020-94b0-e30652c9b721-kube-api-access-w4d7p\") pod \"distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7\" (UID: \"d174a74f-7205-4020-94b0-e30652c9b721\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" Oct 30 17:33:48.483051 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.482950 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz"] Oct 30 17:33:48.662896 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.652484 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d174a74f-7205-4020-94b0-e30652c9b721-dshm\") pod \"distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7\" (UID: \"d174a74f-7205-4020-94b0-e30652c9b721\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" Oct 30 17:33:48.662896 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.658579 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d174a74f-7205-4020-94b0-e30652c9b721-tls-certs\") pod \"distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7\" (UID: \"d174a74f-7205-4020-94b0-e30652c9b721\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" Oct 30 17:33:48.665618 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.665590 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" Oct 30 17:33:48.686121 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.686090 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz"] Oct 30 17:33:48.686429 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.686411 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:33:48.714091 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.710325 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-8d946d-openshift-default-dockercfg-txjzw\"" Oct 30 17:33:48.759145 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.759111 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-workload-certs\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:33:48.759449 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.759413 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-istio-data\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:33:48.759612 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.759593 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d5fb88-e105-459f-a9cd-959eb131c883-istiod-ca-cert\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:33:48.760031 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.760001 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-credential-socket\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:33:48.760222 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.760195 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4d5fb88-e105-459f-a9cd-959eb131c883-istio-token\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:33:48.760355 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.760338 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d5fb88-e105-459f-a9cd-959eb131c883-istio-podinfo\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:33:48.760510 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.760492 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-workload-socket\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:33:48.760661 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.760644 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-istio-envoy\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:33:48.760889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.760851 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkxlr\" (UniqueName: \"kubernetes.io/projected/d4d5fb88-e105-459f-a9cd-959eb131c883-kube-api-access-pkxlr\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:33:48.837017 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.836977 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:48.839208 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.839184 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:33:48.875888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.866983 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-credential-socket\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:33:48.875888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.867063 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4d5fb88-e105-459f-a9cd-959eb131c883-istio-token\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:33:48.875888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.867092 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d5fb88-e105-459f-a9cd-959eb131c883-istio-podinfo\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:33:48.875888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.867141 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-workload-socket\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:33:48.875888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.867189 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-istio-envoy\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:33:48.875888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.867273 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkxlr\" (UniqueName: \"kubernetes.io/projected/d4d5fb88-e105-459f-a9cd-959eb131c883-kube-api-access-pkxlr\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:33:48.875888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.867360 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-workload-certs\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:33:48.875888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.867470 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-istio-data\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:33:48.875888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.867519 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d5fb88-e105-459f-a9cd-959eb131c883-istiod-ca-cert\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:33:48.881817 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.881760 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d5fb88-e105-459f-a9cd-959eb131c883-istiod-ca-cert\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:33:48.882763 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.882723 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-workload-socket\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:33:48.883935 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.883887 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-credential-socket\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:33:48.884276 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.884234 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-istio-data\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:33:48.884728 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.884687 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-workload-certs\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:33:48.950907 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.933745 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d5fb88-e105-459f-a9cd-959eb131c883-istio-podinfo\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:33:48.950907 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.946963 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:48.950907 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:48.948058 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:33:49.012940 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:49.012804 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 17:33:49.051706 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:49.051672 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:33:49.162687 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:49.162657 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:33:49.186495 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:49.186464 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkxlr\" (UniqueName: \"kubernetes.io/projected/d4d5fb88-e105-459f-a9cd-959eb131c883-kube-api-access-pkxlr\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:33:49.191032 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:49.190975 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-istio-envoy\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:33:49.193187 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:49.193154 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 17:33:49.194324 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:49.194300 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:33:49.199878 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:49.195766 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:49.207501 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:49.207470 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:33:49.388884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:49.361561 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 17:33:49.447918 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:49.415950 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:49.780952 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:49.667336 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4d5fb88-e105-459f-a9cd-959eb131c883-istio-token\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:33:49.847889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:49.847419 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:33:49.953269 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:49.953220 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:50.235313 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:50.076269 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 17:33:50.265511 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:50.166262 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:51.161434 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:51.161390 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-222.ec2.internal" podUID="67161c7401d4577fb0911752be0668c2" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:51.250958 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:51.250397 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:51.282012 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:51.281034 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:51.353889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:51.333575 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 17:33:51.381068 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:51.380971 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 17:33:51.419918 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:51.391034 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 17:33:51.419918 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:51.381165 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:51.494118 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:33:51.494083 4965 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.386s" Oct 30 17:33:51.494398 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:51.494377 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:33:51.519888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:51.494604 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:33:51.521046 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:51.521024 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:33:51.541885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:51.528024 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7"] Oct 30 17:33:51.569050 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:51.569012 4965 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:33:51.569348 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:51.569315 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" podUID="ddb82e7c-5038-46b7-937e-c9ae4b12b2c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:51.893887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:51.886336 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:51.921778 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:51.921404 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-xmr9m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:33:51.921778 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:51.921466 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" podUID="d054b35e-8973-4d7b-84e5-6a49ac14bfe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:52.135445 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:52.135402 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Oct 30 17:33:52.338459 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:52.338411 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 17:33:52.367985 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:52.367940 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 17:33:52.393260 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:52.393218 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:52.441594 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:52.441553 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:33:52.441798 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:52.441780 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:33:52.441944 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:52.441928 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:33:52.814600 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:52.801036 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz"] Oct 30 17:33:52.814600 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:52.813523 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:33:52.814600 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:52.813570 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" event={"ID":"d174a74f-7205-4020-94b0-e30652c9b721","Type":"ContainerStarted","Data":"535ac247b14dc551828580b4d956236f623ba1caf15c11fc092cf71970df265f"} Oct 30 17:33:52.814600 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:52.813620 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:33:52.814600 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:52.813655 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:33:52.814600 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:52.813688 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:33:52.814600 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:52.813720 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:33:53.095901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:53.095852 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rh774 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:33:53.096127 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:53.096092 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" podUID="337938ce-7eb2-45bf-a5aa-0ef36cee9444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:53.706941 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:53.703271 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" event={"ID":"d174a74f-7205-4020-94b0-e30652c9b721","Type":"ContainerStarted","Data":"e0ce0b77cde220bc1f335db9d3713db548d8e67d493f6889a0bf14b160c14ca2"} Oct 30 17:33:53.750432 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:53.750385 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" event={"ID":"d4d5fb88-e105-459f-a9cd-959eb131c883","Type":"ContainerStarted","Data":"3dc003052fc409784cd6968b3a1625432af4248d4b15dd368a7618d27e3adda8"} Oct 30 17:33:53.750722 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:53.750704 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" event={"ID":"d4d5fb88-e105-459f-a9cd-959eb131c883","Type":"ContainerStarted","Data":"913ede38f56b5c324b582fb16ffa5fb1c0be9701c8db080e4f70b3da8fd2cf07"} Oct 30 17:33:53.852719 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:53.852630 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" podStartSLOduration=6.8526120729999995 podStartE2EDuration="6.852612073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:53.790483831 +0000 UTC m=+1291.842583764" watchObservedRunningTime="2026-10-30 17:33:53.852612073 +0000 UTC m=+1291.904712007" Oct 30 17:33:53.935700 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:53.935614 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" podStartSLOduration=5.935595162 podStartE2EDuration="5.935595162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:53.872276806 +0000 UTC m=+1291.924376737" watchObservedRunningTime="2026-10-30 17:33:53.935595162 +0000 UTC m=+1291.987695095" Oct 30 17:33:54.019772 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:54.019686 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:33:54.054709 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:54.054671 4965 patch_prober.go:29] interesting pod/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 17:33:54.055010 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:54.054972 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" podUID="d4d5fb88-e105-459f-a9cd-959eb131c883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 17:33:54.666034 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:54.665977 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" Oct 30 17:33:55.013557 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:55.013467 4965 patch_prober.go:29] interesting pod/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 17:33:55.013557 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:55.013520 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" podUID="d4d5fb88-e105-459f-a9cd-959eb131c883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 17:33:56.013991 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:56.013956 4965 patch_prober.go:29] interesting pod/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 17:33:56.014382 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:56.014016 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" podUID="d4d5fb88-e105-459f-a9cd-959eb131c883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 17:33:57.013702 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:57.013663 4965 patch_prober.go:29] interesting pod/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 17:33:57.013904 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:57.013725 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" podUID="d4d5fb88-e105-459f-a9cd-959eb131c883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 17:33:58.013727 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:58.013692 4965 patch_prober.go:29] interesting pod/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 17:33:58.014182 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:58.013743 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" podUID="d4d5fb88-e105-459f-a9cd-959eb131c883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 17:33:58.672115 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:58.672081 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" Oct 30 17:33:58.696730 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:58.696701 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" Oct 30 17:33:58.934215 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:58.934119 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" Oct 30 17:33:59.013695 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:59.013656 4965 patch_prober.go:29] interesting pod/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 17:33:59.013828 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:33:59.013730 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" podUID="d4d5fb88-e105-459f-a9cd-959eb131c883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 17:34:00.013976 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:00.013940 4965 patch_prober.go:29] interesting pod/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 17:34:00.014459 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:00.013999 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" podUID="d4d5fb88-e105-459f-a9cd-959eb131c883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 17:34:01.019695 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:01.019663 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:34:01.020243 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:01.020211 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:34:01.021002 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:01.020981 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 17:34:04.450399 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:04.450309 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f8ac7b58-5dc5d74fc4-kskcv"] Oct 30 17:34:04.481742 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:04.481712 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f8ac7b58-5d5f6b56bc-mt8dc"] Oct 30 17:34:06.069402 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:34:06.069359 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:34:09.450944 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:09.450907 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f8ac7b58-5dc5d74fc4-kskcv" podUID="8239abfe-bed3-4eac-8387-a7eec36f9d8b" containerName="payload-processing" containerID="cri-o://d98719bdc6ee6642de99788db0ad6f9701a5d6f2bb2ccb8e93323b2e6f872dd0" gracePeriod=25 Oct 30 17:34:09.482547 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:09.482516 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f8ac7b58-5d5f6b56bc-mt8dc" podUID="a20e9e32-4b37-4268-b2d9-3e23b1607c52" containerName="payload-pre-processing" containerID="cri-o://83f518eb85b6e5e42f81450a4957b585ab1cdf1371d625a34bb06d68da8809dc" gracePeriod=25 Oct 30 17:34:09.772841 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:09.772816 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f8ac7b58-5dc5d74fc4-kskcv" Oct 30 17:34:09.840242 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:09.840210 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f8ac7b58-5d5f6b56bc-mt8dc" Oct 30 17:34:09.868107 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:09.868078 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a20e9e32-4b37-4268-b2d9-3e23b1607c52-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a20e9e32-4b37-4268-b2d9-3e23b1607c52-plugins-config-volume\") pod \"a20e9e32-4b37-4268-b2d9-3e23b1607c52\" (UID: \"a20e9e32-4b37-4268-b2d9-3e23b1607c52\") " Oct 30 17:34:09.868287 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:09.868202 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a20e9e32-4b37-4268-b2d9-3e23b1607c52-kube-api-access-6k2nf\" (UniqueName: \"kubernetes.io/projected/a20e9e32-4b37-4268-b2d9-3e23b1607c52-kube-api-access-6k2nf\") pod \"a20e9e32-4b37-4268-b2d9-3e23b1607c52\" (UID: \"a20e9e32-4b37-4268-b2d9-3e23b1607c52\") " Oct 30 17:34:09.868359 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:09.868286 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8239abfe-bed3-4eac-8387-a7eec36f9d8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8239abfe-bed3-4eac-8387-a7eec36f9d8b-plugins-config-volume\") pod \"8239abfe-bed3-4eac-8387-a7eec36f9d8b\" (UID: \"8239abfe-bed3-4eac-8387-a7eec36f9d8b\") " Oct 30 17:34:09.868422 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:09.868355 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8239abfe-bed3-4eac-8387-a7eec36f9d8b-kube-api-access-wwhbs\" (UniqueName: \"kubernetes.io/projected/8239abfe-bed3-4eac-8387-a7eec36f9d8b-kube-api-access-wwhbs\") pod \"8239abfe-bed3-4eac-8387-a7eec36f9d8b\" (UID: \"8239abfe-bed3-4eac-8387-a7eec36f9d8b\") " Oct 30 17:34:09.868523 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:09.868500 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a20e9e32-4b37-4268-b2d9-3e23b1607c52-plugins-config-volume" pod "a20e9e32-4b37-4268-b2d9-3e23b1607c52" (UID: "a20e9e32-4b37-4268-b2d9-3e23b1607c52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:09.868793 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:09.868768 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8239abfe-bed3-4eac-8387-a7eec36f9d8b-plugins-config-volume" pod "8239abfe-bed3-4eac-8387-a7eec36f9d8b" (UID: "8239abfe-bed3-4eac-8387-a7eec36f9d8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:09.869176 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:09.869152 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a20e9e32-4b37-4268-b2d9-3e23b1607c52-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:34:09.869268 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:09.869183 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8239abfe-bed3-4eac-8387-a7eec36f9d8b-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:34:09.870696 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:09.870672 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8239abfe-bed3-4eac-8387-a7eec36f9d8b-kube-api-access-wwhbs" pod "8239abfe-bed3-4eac-8387-a7eec36f9d8b" (UID: "8239abfe-bed3-4eac-8387-a7eec36f9d8b"). InnerVolumeSpecName "kube-api-access-wwhbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:09.870827 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:09.870699 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a20e9e32-4b37-4268-b2d9-3e23b1607c52-kube-api-access-6k2nf" pod "a20e9e32-4b37-4268-b2d9-3e23b1607c52" (UID: "a20e9e32-4b37-4268-b2d9-3e23b1607c52"). InnerVolumeSpecName "kube-api-access-6k2nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:09.970897 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:09.970834 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k2nf\" (UniqueName: \"kubernetes.io/projected/a20e9e32-4b37-4268-b2d9-3e23b1607c52-kube-api-access-6k2nf\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:34:09.970897 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:09.970900 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwhbs\" (UniqueName: \"kubernetes.io/projected/8239abfe-bed3-4eac-8387-a7eec36f9d8b-kube-api-access-wwhbs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:34:10.018760 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:10.018678 4965 generic.go:356] "Generic (PLEG): container finished" podID="8239abfe-bed3-4eac-8387-a7eec36f9d8b" containerID="d98719bdc6ee6642de99788db0ad6f9701a5d6f2bb2ccb8e93323b2e6f872dd0" exitCode=0 Oct 30 17:34:10.018760 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:10.018725 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f8ac7b58-5dc5d74fc4-kskcv" Oct 30 17:34:10.018999 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:10.018759 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f8ac7b58-5dc5d74fc4-kskcv" event={"ID":"8239abfe-bed3-4eac-8387-a7eec36f9d8b","Type":"ContainerDied","Data":"d98719bdc6ee6642de99788db0ad6f9701a5d6f2bb2ccb8e93323b2e6f872dd0"} Oct 30 17:34:10.018999 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:10.018816 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f8ac7b58-5dc5d74fc4-kskcv" event={"ID":"8239abfe-bed3-4eac-8387-a7eec36f9d8b","Type":"ContainerDied","Data":"30d9777b4536e72366b14f6bbd972306bee55bcacf6a119e2996073f8d04e2eb"} Oct 30 17:34:10.018999 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:10.018843 4965 scope.go:122] "RemoveContainer" containerID="d98719bdc6ee6642de99788db0ad6f9701a5d6f2bb2ccb8e93323b2e6f872dd0" Oct 30 17:34:10.020242 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:10.020216 4965 generic.go:356] "Generic (PLEG): container finished" podID="a20e9e32-4b37-4268-b2d9-3e23b1607c52" containerID="83f518eb85b6e5e42f81450a4957b585ab1cdf1371d625a34bb06d68da8809dc" exitCode=0 Oct 30 17:34:10.020344 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:10.020256 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f8ac7b58-5d5f6b56bc-mt8dc" Oct 30 17:34:10.020344 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:10.020277 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f8ac7b58-5d5f6b56bc-mt8dc" event={"ID":"a20e9e32-4b37-4268-b2d9-3e23b1607c52","Type":"ContainerDied","Data":"83f518eb85b6e5e42f81450a4957b585ab1cdf1371d625a34bb06d68da8809dc"} Oct 30 17:34:10.020344 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:10.020311 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f8ac7b58-5d5f6b56bc-mt8dc" event={"ID":"a20e9e32-4b37-4268-b2d9-3e23b1607c52","Type":"ContainerDied","Data":"db3a8df94457f846f4d8acef90bca9025bb9c835a5fa8b41b4526317cc4aa0ab"} Oct 30 17:34:10.033704 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:10.033687 4965 scope.go:122] "RemoveContainer" containerID="d98719bdc6ee6642de99788db0ad6f9701a5d6f2bb2ccb8e93323b2e6f872dd0" Oct 30 17:34:10.034040 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:34:10.034020 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d98719bdc6ee6642de99788db0ad6f9701a5d6f2bb2ccb8e93323b2e6f872dd0\": container with ID starting with d98719bdc6ee6642de99788db0ad6f9701a5d6f2bb2ccb8e93323b2e6f872dd0 not found: ID does not exist" containerID="d98719bdc6ee6642de99788db0ad6f9701a5d6f2bb2ccb8e93323b2e6f872dd0" Oct 30 17:34:10.034101 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:10.034047 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d98719bdc6ee6642de99788db0ad6f9701a5d6f2bb2ccb8e93323b2e6f872dd0"} err="failed to get container status \"d98719bdc6ee6642de99788db0ad6f9701a5d6f2bb2ccb8e93323b2e6f872dd0\": rpc error: code = NotFound desc = could not find container \"d98719bdc6ee6642de99788db0ad6f9701a5d6f2bb2ccb8e93323b2e6f872dd0\": container with ID starting with d98719bdc6ee6642de99788db0ad6f9701a5d6f2bb2ccb8e93323b2e6f872dd0 not found: ID does not exist" Oct 30 17:34:10.034101 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:10.034071 4965 scope.go:122] "RemoveContainer" containerID="83f518eb85b6e5e42f81450a4957b585ab1cdf1371d625a34bb06d68da8809dc" Oct 30 17:34:10.043537 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:10.043511 4965 scope.go:122] "RemoveContainer" containerID="83f518eb85b6e5e42f81450a4957b585ab1cdf1371d625a34bb06d68da8809dc" Oct 30 17:34:10.043831 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:34:10.043814 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83f518eb85b6e5e42f81450a4957b585ab1cdf1371d625a34bb06d68da8809dc\": container with ID starting with 83f518eb85b6e5e42f81450a4957b585ab1cdf1371d625a34bb06d68da8809dc not found: ID does not exist" containerID="83f518eb85b6e5e42f81450a4957b585ab1cdf1371d625a34bb06d68da8809dc" Oct 30 17:34:10.043921 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:10.043838 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83f518eb85b6e5e42f81450a4957b585ab1cdf1371d625a34bb06d68da8809dc"} err="failed to get container status \"83f518eb85b6e5e42f81450a4957b585ab1cdf1371d625a34bb06d68da8809dc\": rpc error: code = NotFound desc = could not find container \"83f518eb85b6e5e42f81450a4957b585ab1cdf1371d625a34bb06d68da8809dc\": container with ID starting with 83f518eb85b6e5e42f81450a4957b585ab1cdf1371d625a34bb06d68da8809dc not found: ID does not exist" Oct 30 17:34:10.073014 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:10.072991 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f8ac7b58-5d5f6b56bc-mt8dc"] Oct 30 17:34:10.076958 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:10.076938 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f8ac7b58-5d5f6b56bc-mt8dc"] Oct 30 17:34:10.099755 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:10.099717 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a20e9e32-4b37-4268-b2d9-3e23b1607c52" path="/var/lib/kubelet/pods/a20e9e32-4b37-4268-b2d9-3e23b1607c52/volumes" Oct 30 17:34:10.112903 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:10.112856 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f8ac7b58-5dc5d74fc4-kskcv"] Oct 30 17:34:10.116672 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:10.116649 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f8ac7b58-5dc5d74fc4-kskcv"] Oct 30 17:34:11.497538 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:11.497493 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-84af0117-845ccbf768-mfwg2"] Oct 30 17:34:11.528564 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:11.528530 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-84af0117-687b6d54bc-mn5pk"] Oct 30 17:34:12.100102 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:12.100073 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8239abfe-bed3-4eac-8387-a7eec36f9d8b" path="/var/lib/kubelet/pods/8239abfe-bed3-4eac-8387-a7eec36f9d8b/volumes" Oct 30 17:34:14.237882 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.218765 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk"] Oct 30 17:34:14.237882 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.219775 4965 cpu_manager.go:414] "containerMap: removing container" podUID="a20e9e32-4b37-4268-b2d9-3e23b1607c52" containerName="payload-pre-processing" Oct 30 17:34:14.237882 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.219792 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a20e9e32-4b37-4268-b2d9-3e23b1607c52" containerName="payload-pre-processing" Oct 30 17:34:14.237882 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.219852 4965 cpu_manager.go:414] "containerMap: removing container" podUID="8239abfe-bed3-4eac-8387-a7eec36f9d8b" containerName="payload-processing" Oct 30 17:34:14.237882 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.219876 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8239abfe-bed3-4eac-8387-a7eec36f9d8b" containerName="payload-processing" Oct 30 17:34:14.237882 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.220032 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="8239abfe-bed3-4eac-8387-a7eec36f9d8b" containerName="payload-processing" Oct 30 17:34:14.237882 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.220051 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="a20e9e32-4b37-4268-b2d9-3e23b1607c52" containerName="payload-pre-processing" Oct 30 17:34:14.237882 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.233846 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:14.258461 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.257276 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk"] Oct 30 17:34:14.258461 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.257717 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-86c97ce8-openshift-default-dockercfg-r7k55\"" Oct 30 17:34:14.438413 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.406030 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istio-podinfo\") pod \"e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:14.438413 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.406161 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istiod-ca-cert\") pod \"e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:14.438413 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.406230 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-workload-certs\") pod \"e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:14.438413 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.406267 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-workload-socket\") pod \"e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:14.438413 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.406318 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istio-envoy\") pod \"e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:14.438413 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.406475 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-credential-socket\") pod \"e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:14.438413 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.406549 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcm95\" (UniqueName: \"kubernetes.io/projected/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-kube-api-access-hcm95\") pod \"e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:14.438413 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.406592 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istio-data\") pod \"e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:14.438413 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.406643 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istio-token\") pod \"e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:14.545938 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.524340 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istio-envoy\") pod \"e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:14.545938 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.524508 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-credential-socket\") pod \"e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:14.545938 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.524578 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcm95\" (UniqueName: \"kubernetes.io/projected/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-kube-api-access-hcm95\") pod \"e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:14.545938 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.524620 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istio-data\") pod \"e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:14.545938 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.524669 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istio-token\") pod \"e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:14.545938 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.524756 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istio-podinfo\") pod \"e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:14.545938 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.524833 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istiod-ca-cert\") pod \"e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:14.545938 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.524914 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-workload-certs\") pod \"e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:14.545938 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.524952 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-workload-socket\") pod \"e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:14.545938 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.525384 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-workload-socket\") pod \"e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:14.545938 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.525440 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-credential-socket\") pod \"e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:14.545938 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.525723 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istio-data\") pod \"e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:14.545938 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.525997 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-workload-certs\") pod \"e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:14.565886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.555224 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istiod-ca-cert\") pod \"e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:14.573887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.568522 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istio-podinfo\") pod \"e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:14.578969 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.578634 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istio-token\") pod \"e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:14.597895 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.595521 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istio-envoy\") pod \"e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:14.597895 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.596220 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcm95\" (UniqueName: \"kubernetes.io/projected/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-kube-api-access-hcm95\") pod \"e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:14.868815 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:14.868783 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:15.408684 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:15.408629 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:34:15.409322 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:15.409290 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:34:15.409488 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:15.409470 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:34:15.434882 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:15.429447 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk"] Oct 30 17:34:16.117001 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:16.116958 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" event={"ID":"dd97b0c7-6847-48fe-b7bd-06cb3b78935c","Type":"ContainerStarted","Data":"da3a8c85785d77a58022c01a7086b6e18addda10b74efc7a3fb607d619edf4f5"} Oct 30 17:34:16.117187 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:16.117019 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" event={"ID":"dd97b0c7-6847-48fe-b7bd-06cb3b78935c","Type":"ContainerStarted","Data":"3b143e33ead3f55b220b9d407f92f2824c44f2c37415ecd998bd6851f3541117"} Oct 30 17:34:16.173680 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:16.173626 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" podStartSLOduration=2.17361291 podStartE2EDuration="2.17361291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:34:16.173038397 +0000 UTC m=+1314.225138342" watchObservedRunningTime="2026-10-30 17:34:16.17361291 +0000 UTC m=+1314.225712840" Oct 30 17:34:16.498111 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:16.498010 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-84af0117-845ccbf768-mfwg2" podUID="a4197c2a-53e7-436e-85ab-fd1417088a29" containerName="payload-processing" containerID="cri-o://5015d4432315dcb769ed6870f1a9e0ef62b79f61beef2c456e62ce6535f98d5b" gracePeriod=25 Oct 30 17:34:16.529307 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:16.529276 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-84af0117-687b6d54bc-mn5pk" podUID="e615e20e-ad32-4c76-b1aa-4b447b019699" containerName="payload-pre-processing" containerID="cri-o://23252fa4cebb2cef94a3c345367e9750346da7f71148561ad12d0e231348a3fd" gracePeriod=25 Oct 30 17:34:16.835929 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:16.835904 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-84af0117-845ccbf768-mfwg2" Oct 30 17:34:16.853569 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:16.853542 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-84af0117-687b6d54bc-mn5pk" Oct 30 17:34:16.869816 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:16.869797 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:16.876410 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:16.876375 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:16.991651 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:16.991621 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e615e20e-ad32-4c76-b1aa-4b447b019699-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e615e20e-ad32-4c76-b1aa-4b447b019699-plugins-config-volume\") pod \"e615e20e-ad32-4c76-b1aa-4b447b019699\" (UID: \"e615e20e-ad32-4c76-b1aa-4b447b019699\") " Oct 30 17:34:16.991836 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:16.991755 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4197c2a-53e7-436e-85ab-fd1417088a29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4197c2a-53e7-436e-85ab-fd1417088a29-plugins-config-volume\") pod \"a4197c2a-53e7-436e-85ab-fd1417088a29\" (UID: \"a4197c2a-53e7-436e-85ab-fd1417088a29\") " Oct 30 17:34:16.991836 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:16.991792 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4197c2a-53e7-436e-85ab-fd1417088a29-kube-api-access-fzlv5\" (UniqueName: \"kubernetes.io/projected/a4197c2a-53e7-436e-85ab-fd1417088a29-kube-api-access-fzlv5\") pod \"a4197c2a-53e7-436e-85ab-fd1417088a29\" (UID: \"a4197c2a-53e7-436e-85ab-fd1417088a29\") " Oct 30 17:34:16.992022 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:16.991959 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e615e20e-ad32-4c76-b1aa-4b447b019699-kube-api-access-gt9cg\" (UniqueName: \"kubernetes.io/projected/e615e20e-ad32-4c76-b1aa-4b447b019699-kube-api-access-gt9cg\") pod \"e615e20e-ad32-4c76-b1aa-4b447b019699\" (UID: \"e615e20e-ad32-4c76-b1aa-4b447b019699\") " Oct 30 17:34:16.992121 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:16.992098 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e615e20e-ad32-4c76-b1aa-4b447b019699-plugins-config-volume" pod "e615e20e-ad32-4c76-b1aa-4b447b019699" (UID: "e615e20e-ad32-4c76-b1aa-4b447b019699"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:16.992183 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:16.992140 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4197c2a-53e7-436e-85ab-fd1417088a29-plugins-config-volume" pod "a4197c2a-53e7-436e-85ab-fd1417088a29" (UID: "a4197c2a-53e7-436e-85ab-fd1417088a29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:16.992455 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:16.992435 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e615e20e-ad32-4c76-b1aa-4b447b019699-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:34:16.992515 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:16.992460 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4197c2a-53e7-436e-85ab-fd1417088a29-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:34:16.994071 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:16.994048 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e615e20e-ad32-4c76-b1aa-4b447b019699-kube-api-access-gt9cg" pod "e615e20e-ad32-4c76-b1aa-4b447b019699" (UID: "e615e20e-ad32-4c76-b1aa-4b447b019699"). InnerVolumeSpecName "kube-api-access-gt9cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:16.994202 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:16.994181 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4197c2a-53e7-436e-85ab-fd1417088a29-kube-api-access-fzlv5" pod "a4197c2a-53e7-436e-85ab-fd1417088a29" (UID: "a4197c2a-53e7-436e-85ab-fd1417088a29"). InnerVolumeSpecName "kube-api-access-fzlv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:17.093594 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:17.093566 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzlv5\" (UniqueName: \"kubernetes.io/projected/a4197c2a-53e7-436e-85ab-fd1417088a29-kube-api-access-fzlv5\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:34:17.093594 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:17.093591 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt9cg\" (UniqueName: \"kubernetes.io/projected/e615e20e-ad32-4c76-b1aa-4b447b019699-kube-api-access-gt9cg\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:34:17.199889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:17.191679 4965 generic.go:356] "Generic (PLEG): container finished" podID="e615e20e-ad32-4c76-b1aa-4b447b019699" containerID="23252fa4cebb2cef94a3c345367e9750346da7f71148561ad12d0e231348a3fd" exitCode=0 Oct 30 17:34:17.199889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:17.191999 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-84af0117-687b6d54bc-mn5pk" event={"ID":"e615e20e-ad32-4c76-b1aa-4b447b019699","Type":"ContainerDied","Data":"23252fa4cebb2cef94a3c345367e9750346da7f71148561ad12d0e231348a3fd"} Oct 30 17:34:17.199889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:17.192061 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-84af0117-687b6d54bc-mn5pk" event={"ID":"e615e20e-ad32-4c76-b1aa-4b447b019699","Type":"ContainerDied","Data":"b72dfc86ac1974c3ff62d2f7b6502bc5e443c842e74643c94bb2cd2dc349f15b"} Oct 30 17:34:17.199889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:17.192091 4965 scope.go:122] "RemoveContainer" containerID="23252fa4cebb2cef94a3c345367e9750346da7f71148561ad12d0e231348a3fd" Oct 30 17:34:17.199889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:17.192274 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-84af0117-687b6d54bc-mn5pk" Oct 30 17:34:17.223266 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:17.220032 4965 generic.go:356] "Generic (PLEG): container finished" podID="a4197c2a-53e7-436e-85ab-fd1417088a29" containerID="5015d4432315dcb769ed6870f1a9e0ef62b79f61beef2c456e62ce6535f98d5b" exitCode=0 Oct 30 17:34:17.223266 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:17.221546 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-84af0117-845ccbf768-mfwg2" Oct 30 17:34:17.223266 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:17.222116 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-84af0117-845ccbf768-mfwg2" event={"ID":"a4197c2a-53e7-436e-85ab-fd1417088a29","Type":"ContainerDied","Data":"5015d4432315dcb769ed6870f1a9e0ef62b79f61beef2c456e62ce6535f98d5b"} Oct 30 17:34:17.223266 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:17.222188 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:17.223266 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:17.222217 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-84af0117-845ccbf768-mfwg2" event={"ID":"a4197c2a-53e7-436e-85ab-fd1417088a29","Type":"ContainerDied","Data":"59e39a4bfe764e4e3b3cbac87b17b9778f1ccfbb9e89db131c9a2c013188ed78"} Oct 30 17:34:17.233063 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:17.227096 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:34:17.264733 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:17.264707 4965 scope.go:122] "RemoveContainer" containerID="23252fa4cebb2cef94a3c345367e9750346da7f71148561ad12d0e231348a3fd" Oct 30 17:34:17.265526 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:34:17.265504 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23252fa4cebb2cef94a3c345367e9750346da7f71148561ad12d0e231348a3fd\": container with ID starting with 23252fa4cebb2cef94a3c345367e9750346da7f71148561ad12d0e231348a3fd not found: ID does not exist" containerID="23252fa4cebb2cef94a3c345367e9750346da7f71148561ad12d0e231348a3fd" Oct 30 17:34:17.265647 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:17.265532 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23252fa4cebb2cef94a3c345367e9750346da7f71148561ad12d0e231348a3fd"} err="failed to get container status \"23252fa4cebb2cef94a3c345367e9750346da7f71148561ad12d0e231348a3fd\": rpc error: code = NotFound desc = could not find container \"23252fa4cebb2cef94a3c345367e9750346da7f71148561ad12d0e231348a3fd\": container with ID starting with 23252fa4cebb2cef94a3c345367e9750346da7f71148561ad12d0e231348a3fd not found: ID does not exist" Oct 30 17:34:17.265647 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:17.265549 4965 scope.go:122] "RemoveContainer" containerID="5015d4432315dcb769ed6870f1a9e0ef62b79f61beef2c456e62ce6535f98d5b" Oct 30 17:34:17.275365 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:17.275344 4965 scope.go:122] "RemoveContainer" containerID="5015d4432315dcb769ed6870f1a9e0ef62b79f61beef2c456e62ce6535f98d5b" Oct 30 17:34:17.275644 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:34:17.275627 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5015d4432315dcb769ed6870f1a9e0ef62b79f61beef2c456e62ce6535f98d5b\": container with ID starting with 5015d4432315dcb769ed6870f1a9e0ef62b79f61beef2c456e62ce6535f98d5b not found: ID does not exist" containerID="5015d4432315dcb769ed6870f1a9e0ef62b79f61beef2c456e62ce6535f98d5b" Oct 30 17:34:17.275691 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:17.275651 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5015d4432315dcb769ed6870f1a9e0ef62b79f61beef2c456e62ce6535f98d5b"} err="failed to get container status \"5015d4432315dcb769ed6870f1a9e0ef62b79f61beef2c456e62ce6535f98d5b\": rpc error: code = NotFound desc = could not find container \"5015d4432315dcb769ed6870f1a9e0ef62b79f61beef2c456e62ce6535f98d5b\": container with ID starting with 5015d4432315dcb769ed6870f1a9e0ef62b79f61beef2c456e62ce6535f98d5b not found: ID does not exist" Oct 30 17:34:17.339943 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:17.336795 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-84af0117-845ccbf768-mfwg2"] Oct 30 17:34:17.345073 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:17.345010 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-84af0117-845ccbf768-mfwg2"] Oct 30 17:34:17.369653 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:17.369622 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-84af0117-687b6d54bc-mn5pk"] Oct 30 17:34:17.373918 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:17.373888 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-84af0117-687b6d54bc-mn5pk"] Oct 30 17:34:18.106647 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:18.106613 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4197c2a-53e7-436e-85ab-fd1417088a29" path="/var/lib/kubelet/pods/a4197c2a-53e7-436e-85ab-fd1417088a29/volumes" Oct 30 17:34:18.107240 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:18.107223 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e615e20e-ad32-4c76-b1aa-4b447b019699" path="/var/lib/kubelet/pods/e615e20e-ad32-4c76-b1aa-4b447b019699/volumes" Oct 30 17:34:19.296889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:19.295673 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:34:21.197929 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:21.188947 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-15a954ad-6c5fb7c4c6-vht7j"] Oct 30 17:34:21.210049 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:21.210014 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-15a954ad-8b6c47b87-76tsj"] Oct 30 17:34:23.183888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:23.174002 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:34:23.490666 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:23.490569 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:34:24.242217 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:24.241967 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:34:24.862887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:24.854983 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" podUID="d174a74f-7205-4020-94b0-e30652c9b721" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:34:24.862887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:24.858598 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2"] Oct 30 17:34:24.862887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:24.858832 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" podUID="52968197-826b-42aa-8627-6287ec75aeed" containerName="istio-proxy" containerID="cri-o://89db0fcb3c90155b50f97695c029fef9179d59b03667925d93f8c6608290b838" gracePeriod=30 Oct 30 17:34:24.984901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:24.939038 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:34:25.166936 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:25.166641 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:34:25.269279 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:25.243768 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:34:25.269279 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:25.244473 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:34:25.273024 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:25.272984 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded" Oct 30 17:34:25.379212 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:25.379167 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:34:25.445684 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:25.445575 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:34:25.504931 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:25.446933 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:34:25.642200 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:25.642167 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:34:26.190054 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:26.190003 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-15a954ad-6c5fb7c4c6-vht7j" podUID="cf726f6a-62b3-4030-9860-58f0fedbf57c" containerName="payload-processing" containerID="cri-o://302359dea91cacc8001f6c0b6cf8af6bc2a7fd474e77c9f98521ce14c4f28ad7" gracePeriod=25 Oct 30 17:34:26.211033 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:26.210994 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-15a954ad-8b6c47b87-76tsj" podUID="5693cca8-8479-4cda-b509-2ee15ef9a3b1" containerName="payload-pre-processing" containerID="cri-o://cf2ce50e0db13122d3f10170320d1d783fc058b28c68afd4c250bb55f63f0b2d" gracePeriod=25 Oct 30 17:34:26.608417 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:26.608388 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-15a954ad-6c5fb7c4c6-vht7j" Oct 30 17:34:26.633468 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:26.633443 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-15a954ad-8b6c47b87-76tsj" Oct 30 17:34:26.654321 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:26.654297 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5693cca8-8479-4cda-b509-2ee15ef9a3b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5693cca8-8479-4cda-b509-2ee15ef9a3b1-plugins-config-volume\") pod \"5693cca8-8479-4cda-b509-2ee15ef9a3b1\" (UID: \"5693cca8-8479-4cda-b509-2ee15ef9a3b1\") " Oct 30 17:34:26.654470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:26.654367 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf726f6a-62b3-4030-9860-58f0fedbf57c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf726f6a-62b3-4030-9860-58f0fedbf57c-plugins-config-volume\") pod \"cf726f6a-62b3-4030-9860-58f0fedbf57c\" (UID: \"cf726f6a-62b3-4030-9860-58f0fedbf57c\") " Oct 30 17:34:26.654470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:26.654393 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf726f6a-62b3-4030-9860-58f0fedbf57c-kube-api-access-hpnxp\" (UniqueName: \"kubernetes.io/projected/cf726f6a-62b3-4030-9860-58f0fedbf57c-kube-api-access-hpnxp\") pod \"cf726f6a-62b3-4030-9860-58f0fedbf57c\" (UID: \"cf726f6a-62b3-4030-9860-58f0fedbf57c\") " Oct 30 17:34:26.654592 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:26.654487 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5693cca8-8479-4cda-b509-2ee15ef9a3b1-kube-api-access-bktzv\" (UniqueName: \"kubernetes.io/projected/5693cca8-8479-4cda-b509-2ee15ef9a3b1-kube-api-access-bktzv\") pod \"5693cca8-8479-4cda-b509-2ee15ef9a3b1\" (UID: \"5693cca8-8479-4cda-b509-2ee15ef9a3b1\") " Oct 30 17:34:26.654736 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:26.654706 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5693cca8-8479-4cda-b509-2ee15ef9a3b1-plugins-config-volume" pod "5693cca8-8479-4cda-b509-2ee15ef9a3b1" (UID: "5693cca8-8479-4cda-b509-2ee15ef9a3b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:26.654805 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:26.654771 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf726f6a-62b3-4030-9860-58f0fedbf57c-plugins-config-volume" pod "cf726f6a-62b3-4030-9860-58f0fedbf57c" (UID: "cf726f6a-62b3-4030-9860-58f0fedbf57c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:26.655243 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:26.655221 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5693cca8-8479-4cda-b509-2ee15ef9a3b1-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:34:26.655243 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:26.655242 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf726f6a-62b3-4030-9860-58f0fedbf57c-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:34:26.656669 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:26.656639 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5693cca8-8479-4cda-b509-2ee15ef9a3b1-kube-api-access-bktzv" pod "5693cca8-8479-4cda-b509-2ee15ef9a3b1" (UID: "5693cca8-8479-4cda-b509-2ee15ef9a3b1"). InnerVolumeSpecName "kube-api-access-bktzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:26.656767 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:26.656671 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf726f6a-62b3-4030-9860-58f0fedbf57c-kube-api-access-hpnxp" pod "cf726f6a-62b3-4030-9860-58f0fedbf57c" (UID: "cf726f6a-62b3-4030-9860-58f0fedbf57c"). InnerVolumeSpecName "kube-api-access-hpnxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:26.802724 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:26.779653 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bktzv\" (UniqueName: \"kubernetes.io/projected/5693cca8-8479-4cda-b509-2ee15ef9a3b1-kube-api-access-bktzv\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:34:26.927353 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:26.927310 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpnxp\" (UniqueName: \"kubernetes.io/projected/cf726f6a-62b3-4030-9860-58f0fedbf57c-kube-api-access-hpnxp\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:34:27.240738 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:27.240708 4965 generic.go:356] "Generic (PLEG): container finished" podID="cf726f6a-62b3-4030-9860-58f0fedbf57c" containerID="302359dea91cacc8001f6c0b6cf8af6bc2a7fd474e77c9f98521ce14c4f28ad7" exitCode=0 Oct 30 17:34:27.240951 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:27.240753 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-15a954ad-6c5fb7c4c6-vht7j" Oct 30 17:34:27.240951 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:27.240797 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-15a954ad-6c5fb7c4c6-vht7j" event={"ID":"cf726f6a-62b3-4030-9860-58f0fedbf57c","Type":"ContainerDied","Data":"302359dea91cacc8001f6c0b6cf8af6bc2a7fd474e77c9f98521ce14c4f28ad7"} Oct 30 17:34:27.240951 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:27.240845 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-15a954ad-6c5fb7c4c6-vht7j" event={"ID":"cf726f6a-62b3-4030-9860-58f0fedbf57c","Type":"ContainerDied","Data":"ee0eb95b565c855bcd2e8b913dfb15503b8059693360563144772a133108f016"} Oct 30 17:34:27.240951 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:27.240900 4965 scope.go:122] "RemoveContainer" containerID="302359dea91cacc8001f6c0b6cf8af6bc2a7fd474e77c9f98521ce14c4f28ad7" Oct 30 17:34:27.242542 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:27.242518 4965 generic.go:356] "Generic (PLEG): container finished" podID="5693cca8-8479-4cda-b509-2ee15ef9a3b1" containerID="cf2ce50e0db13122d3f10170320d1d783fc058b28c68afd4c250bb55f63f0b2d" exitCode=0 Oct 30 17:34:27.242672 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:27.242588 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-15a954ad-8b6c47b87-76tsj" Oct 30 17:34:27.242672 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:27.242608 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-15a954ad-8b6c47b87-76tsj" event={"ID":"5693cca8-8479-4cda-b509-2ee15ef9a3b1","Type":"ContainerDied","Data":"cf2ce50e0db13122d3f10170320d1d783fc058b28c68afd4c250bb55f63f0b2d"} Oct 30 17:34:27.242765 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:27.242671 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-15a954ad-8b6c47b87-76tsj" event={"ID":"5693cca8-8479-4cda-b509-2ee15ef9a3b1","Type":"ContainerDied","Data":"9bb26f5548ce50fa919e628e4ba2551f4be095d506dc020df2600dc062dd8de6"} Oct 30 17:34:27.251811 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:27.251792 4965 scope.go:122] "RemoveContainer" containerID="302359dea91cacc8001f6c0b6cf8af6bc2a7fd474e77c9f98521ce14c4f28ad7" Oct 30 17:34:27.252184 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:34:27.252158 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"302359dea91cacc8001f6c0b6cf8af6bc2a7fd474e77c9f98521ce14c4f28ad7\": container with ID starting with 302359dea91cacc8001f6c0b6cf8af6bc2a7fd474e77c9f98521ce14c4f28ad7 not found: ID does not exist" containerID="302359dea91cacc8001f6c0b6cf8af6bc2a7fd474e77c9f98521ce14c4f28ad7" Oct 30 17:34:27.252276 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:27.252193 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"302359dea91cacc8001f6c0b6cf8af6bc2a7fd474e77c9f98521ce14c4f28ad7"} err="failed to get container status \"302359dea91cacc8001f6c0b6cf8af6bc2a7fd474e77c9f98521ce14c4f28ad7\": rpc error: code = NotFound desc = could not find container \"302359dea91cacc8001f6c0b6cf8af6bc2a7fd474e77c9f98521ce14c4f28ad7\": container with ID starting with 302359dea91cacc8001f6c0b6cf8af6bc2a7fd474e77c9f98521ce14c4f28ad7 not found: ID does not exist" Oct 30 17:34:27.252276 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:27.252216 4965 scope.go:122] "RemoveContainer" containerID="cf2ce50e0db13122d3f10170320d1d783fc058b28c68afd4c250bb55f63f0b2d" Oct 30 17:34:27.287288 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:27.287263 4965 scope.go:122] "RemoveContainer" containerID="cf2ce50e0db13122d3f10170320d1d783fc058b28c68afd4c250bb55f63f0b2d" Oct 30 17:34:27.287595 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:34:27.287575 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf2ce50e0db13122d3f10170320d1d783fc058b28c68afd4c250bb55f63f0b2d\": container with ID starting with cf2ce50e0db13122d3f10170320d1d783fc058b28c68afd4c250bb55f63f0b2d not found: ID does not exist" containerID="cf2ce50e0db13122d3f10170320d1d783fc058b28c68afd4c250bb55f63f0b2d" Oct 30 17:34:27.287667 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:27.287603 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf2ce50e0db13122d3f10170320d1d783fc058b28c68afd4c250bb55f63f0b2d"} err="failed to get container status \"cf2ce50e0db13122d3f10170320d1d783fc058b28c68afd4c250bb55f63f0b2d\": rpc error: code = NotFound desc = could not find container \"cf2ce50e0db13122d3f10170320d1d783fc058b28c68afd4c250bb55f63f0b2d\": container with ID starting with cf2ce50e0db13122d3f10170320d1d783fc058b28c68afd4c250bb55f63f0b2d not found: ID does not exist" Oct 30 17:34:27.295440 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:27.295380 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-15a954ad-8b6c47b87-76tsj"] Oct 30 17:34:27.307212 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:27.307188 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-15a954ad-8b6c47b87-76tsj"] Oct 30 17:34:27.341812 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:27.341781 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-15a954ad-6c5fb7c4c6-vht7j"] Oct 30 17:34:27.345348 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:27.345319 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-15a954ad-6c5fb7c4c6-vht7j"] Oct 30 17:34:28.099990 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:28.099957 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5693cca8-8479-4cda-b509-2ee15ef9a3b1" path="/var/lib/kubelet/pods/5693cca8-8479-4cda-b509-2ee15ef9a3b1/volumes" Oct 30 17:34:28.100490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:28.100475 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf726f6a-62b3-4030-9860-58f0fedbf57c" path="/var/lib/kubelet/pods/cf726f6a-62b3-4030-9860-58f0fedbf57c/volumes" Oct 30 17:34:33.556413 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.556388 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:34:33.595473 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.595445 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh"] Oct 30 17:34:33.596124 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.596108 4965 cpu_manager.go:414] "containerMap: removing container" podUID="52968197-826b-42aa-8627-6287ec75aeed" containerName="istio-proxy" Oct 30 17:34:33.596124 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.596123 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52968197-826b-42aa-8627-6287ec75aeed" containerName="istio-proxy" Oct 30 17:34:33.596239 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.596146 4965 cpu_manager.go:414] "containerMap: removing container" podUID="cf726f6a-62b3-4030-9860-58f0fedbf57c" containerName="payload-processing" Oct 30 17:34:33.596239 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.596154 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf726f6a-62b3-4030-9860-58f0fedbf57c" containerName="payload-processing" Oct 30 17:34:33.596239 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.596172 4965 cpu_manager.go:414] "containerMap: removing container" podUID="5693cca8-8479-4cda-b509-2ee15ef9a3b1" containerName="payload-pre-processing" Oct 30 17:34:33.596239 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.596177 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5693cca8-8479-4cda-b509-2ee15ef9a3b1" containerName="payload-pre-processing" Oct 30 17:34:33.596239 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.596193 4965 cpu_manager.go:414] "containerMap: removing container" podUID="e615e20e-ad32-4c76-b1aa-4b447b019699" containerName="payload-pre-processing" Oct 30 17:34:33.596239 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.596198 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e615e20e-ad32-4c76-b1aa-4b447b019699" containerName="payload-pre-processing" Oct 30 17:34:33.596239 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.596226 4965 cpu_manager.go:414] "containerMap: removing container" podUID="a4197c2a-53e7-436e-85ab-fd1417088a29" containerName="payload-processing" Oct 30 17:34:33.596239 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.596231 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4197c2a-53e7-436e-85ab-fd1417088a29" containerName="payload-processing" Oct 30 17:34:33.596468 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.596326 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf726f6a-62b3-4030-9860-58f0fedbf57c" containerName="payload-processing" Oct 30 17:34:33.596468 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.596339 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="5693cca8-8479-4cda-b509-2ee15ef9a3b1" containerName="payload-pre-processing" Oct 30 17:34:33.596468 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.596348 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="e615e20e-ad32-4c76-b1aa-4b447b019699" containerName="payload-pre-processing" Oct 30 17:34:33.596468 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.596355 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4197c2a-53e7-436e-85ab-fd1417088a29" containerName="payload-processing" Oct 30 17:34:33.596468 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.596361 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="52968197-826b-42aa-8627-6287ec75aeed" containerName="istio-proxy" Oct 30 17:34:33.598704 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.598687 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh" Oct 30 17:34:33.625999 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.625972 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-86c97ce8-serving-cert\"" Oct 30 17:34:33.630888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.630844 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh"] Oct 30 17:34:33.658276 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.658110 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52968197-826b-42aa-8627-6287ec75aeed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52968197-826b-42aa-8627-6287ec75aeed-istiod-ca-cert\") pod \"52968197-826b-42aa-8627-6287ec75aeed\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " Oct 30 17:34:33.658276 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.658237 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-istio-envoy\") pod \"52968197-826b-42aa-8627-6287ec75aeed\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " Oct 30 17:34:33.658276 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.658274 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52968197-826b-42aa-8627-6287ec75aeed-istio-token\" (UniqueName: \"kubernetes.io/projected/52968197-826b-42aa-8627-6287ec75aeed-istio-token\") pod \"52968197-826b-42aa-8627-6287ec75aeed\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " Oct 30 17:34:33.658563 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.658354 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52968197-826b-42aa-8627-6287ec75aeed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52968197-826b-42aa-8627-6287ec75aeed-istio-podinfo\") pod \"52968197-826b-42aa-8627-6287ec75aeed\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " Oct 30 17:34:33.658563 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.658422 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-workload-certs\") pod \"52968197-826b-42aa-8627-6287ec75aeed\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " Oct 30 17:34:33.658563 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.658491 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-credential-socket\") pod \"52968197-826b-42aa-8627-6287ec75aeed\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " Oct 30 17:34:33.658563 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.658531 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52968197-826b-42aa-8627-6287ec75aeed-istiod-ca-cert" pod "52968197-826b-42aa-8627-6287ec75aeed" (UID: "52968197-826b-42aa-8627-6287ec75aeed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:33.658563 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.658561 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-istio-data\") pod \"52968197-826b-42aa-8627-6287ec75aeed\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " Oct 30 17:34:33.658826 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.658597 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-workload-socket\") pod \"52968197-826b-42aa-8627-6287ec75aeed\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " Oct 30 17:34:33.658826 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.658689 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52968197-826b-42aa-8627-6287ec75aeed-kube-api-access-nqv7q\" (UniqueName: \"kubernetes.io/projected/52968197-826b-42aa-8627-6287ec75aeed-kube-api-access-nqv7q\") pod \"52968197-826b-42aa-8627-6287ec75aeed\" (UID: \"52968197-826b-42aa-8627-6287ec75aeed\") " Oct 30 17:34:33.659653 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.659473 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-credential-socket" pod "52968197-826b-42aa-8627-6287ec75aeed" (UID: "52968197-826b-42aa-8627-6287ec75aeed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:34:33.659653 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.659626 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:34:33.659653 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.659635 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-istio-data" pod "52968197-826b-42aa-8627-6287ec75aeed" (UID: "52968197-826b-42aa-8627-6287ec75aeed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:34:33.659932 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.659645 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52968197-826b-42aa-8627-6287ec75aeed-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:34:33.659932 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.659829 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-workload-certs" pod "52968197-826b-42aa-8627-6287ec75aeed" (UID: "52968197-826b-42aa-8627-6287ec75aeed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:34:33.659932 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.659887 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-workload-socket" pod "52968197-826b-42aa-8627-6287ec75aeed" (UID: "52968197-826b-42aa-8627-6287ec75aeed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:34:33.662572 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.662227 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52968197-826b-42aa-8627-6287ec75aeed-kube-api-access-nqv7q" pod "52968197-826b-42aa-8627-6287ec75aeed" (UID: "52968197-826b-42aa-8627-6287ec75aeed"). InnerVolumeSpecName "kube-api-access-nqv7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:33.667591 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.667566 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52968197-826b-42aa-8627-6287ec75aeed-istio-token" pod "52968197-826b-42aa-8627-6287ec75aeed" (UID: "52968197-826b-42aa-8627-6287ec75aeed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:33.668242 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.668201 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52968197-826b-42aa-8627-6287ec75aeed-istio-podinfo" pod "52968197-826b-42aa-8627-6287ec75aeed" (UID: "52968197-826b-42aa-8627-6287ec75aeed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:34:33.670114 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.670089 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-istio-envoy" pod "52968197-826b-42aa-8627-6287ec75aeed" (UID: "52968197-826b-42aa-8627-6287ec75aeed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:34:33.761510 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.761477 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba96a20c-84fa-4a58-916d-0dec80305088-maas-api-tls\") pod \"maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh\" (UID: \"ba96a20c-84fa-4a58-916d-0dec80305088\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh" Oct 30 17:34:33.761671 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.761531 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba96a20c-84fa-4a58-916d-0dec80305088-openshift-service-ca\") pod \"maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh\" (UID: \"ba96a20c-84fa-4a58-916d-0dec80305088\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh" Oct 30 17:34:33.761671 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.761595 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba96a20c-84fa-4a58-916d-0dec80305088-metrics-certs\") pod \"maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh\" (UID: \"ba96a20c-84fa-4a58-916d-0dec80305088\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh" Oct 30 17:34:33.761809 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.761784 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7cgk\" (UniqueName: \"kubernetes.io/projected/ba96a20c-84fa-4a58-916d-0dec80305088-kube-api-access-v7cgk\") pod \"maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh\" (UID: \"ba96a20c-84fa-4a58-916d-0dec80305088\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh" Oct 30 17:34:33.761943 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.761930 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:34:33.762001 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.761945 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:34:33.762001 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.761954 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqv7q\" (UniqueName: \"kubernetes.io/projected/52968197-826b-42aa-8627-6287ec75aeed-kube-api-access-nqv7q\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:34:33.762001 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.761964 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:34:33.762001 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.761973 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52968197-826b-42aa-8627-6287ec75aeed-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:34:33.762001 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.761980 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52968197-826b-42aa-8627-6287ec75aeed-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:34:33.762001 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.761988 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52968197-826b-42aa-8627-6287ec75aeed-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:34:33.887887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.884232 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7cgk\" (UniqueName: \"kubernetes.io/projected/ba96a20c-84fa-4a58-916d-0dec80305088-kube-api-access-v7cgk\") pod \"maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh\" (UID: \"ba96a20c-84fa-4a58-916d-0dec80305088\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh" Oct 30 17:34:33.887887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.884418 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba96a20c-84fa-4a58-916d-0dec80305088-maas-api-tls\") pod \"maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh\" (UID: \"ba96a20c-84fa-4a58-916d-0dec80305088\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh" Oct 30 17:34:33.887887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.884494 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba96a20c-84fa-4a58-916d-0dec80305088-openshift-service-ca\") pod \"maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh\" (UID: \"ba96a20c-84fa-4a58-916d-0dec80305088\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh" Oct 30 17:34:33.887887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.884538 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba96a20c-84fa-4a58-916d-0dec80305088-metrics-certs\") pod \"maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh\" (UID: \"ba96a20c-84fa-4a58-916d-0dec80305088\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh" Oct 30 17:34:33.910886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.892252 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba96a20c-84fa-4a58-916d-0dec80305088-openshift-service-ca\") pod \"maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh\" (UID: \"ba96a20c-84fa-4a58-916d-0dec80305088\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh" Oct 30 17:34:33.910886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.897454 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba96a20c-84fa-4a58-916d-0dec80305088-metrics-certs\") pod \"maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh\" (UID: \"ba96a20c-84fa-4a58-916d-0dec80305088\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh" Oct 30 17:34:33.910886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.897538 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba96a20c-84fa-4a58-916d-0dec80305088-maas-api-tls\") pod \"maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh\" (UID: \"ba96a20c-84fa-4a58-916d-0dec80305088\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh" Oct 30 17:34:33.939085 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:33.932296 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7cgk\" (UniqueName: \"kubernetes.io/projected/ba96a20c-84fa-4a58-916d-0dec80305088-kube-api-access-v7cgk\") pod \"maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh\" (UID: \"ba96a20c-84fa-4a58-916d-0dec80305088\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh" Oct 30 17:34:34.211438 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:34.211358 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh" Oct 30 17:34:34.304912 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:34.304648 4965 generic.go:356] "Generic (PLEG): container finished" podID="52968197-826b-42aa-8627-6287ec75aeed" containerID="89db0fcb3c90155b50f97695c029fef9179d59b03667925d93f8c6608290b838" exitCode=0 Oct 30 17:34:34.304912 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:34.304893 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" event={"ID":"52968197-826b-42aa-8627-6287ec75aeed","Type":"ContainerDied","Data":"89db0fcb3c90155b50f97695c029fef9179d59b03667925d93f8c6608290b838"} Oct 30 17:34:34.305149 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:34.304948 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" event={"ID":"52968197-826b-42aa-8627-6287ec75aeed","Type":"ContainerDied","Data":"518d74a834e222ad3641b0eefc9813b48e797267d28c38443b92449bdbb4a717"} Oct 30 17:34:34.305149 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:34.304977 4965 scope.go:122] "RemoveContainer" containerID="89db0fcb3c90155b50f97695c029fef9179d59b03667925d93f8c6608290b838" Oct 30 17:34:34.305149 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:34.305133 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2" Oct 30 17:34:34.327185 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:34.327048 4965 scope.go:122] "RemoveContainer" containerID="89db0fcb3c90155b50f97695c029fef9179d59b03667925d93f8c6608290b838" Oct 30 17:34:34.327649 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:34:34.327615 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89db0fcb3c90155b50f97695c029fef9179d59b03667925d93f8c6608290b838\": container with ID starting with 89db0fcb3c90155b50f97695c029fef9179d59b03667925d93f8c6608290b838 not found: ID does not exist" containerID="89db0fcb3c90155b50f97695c029fef9179d59b03667925d93f8c6608290b838" Oct 30 17:34:34.327762 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:34.327679 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89db0fcb3c90155b50f97695c029fef9179d59b03667925d93f8c6608290b838"} err="failed to get container status \"89db0fcb3c90155b50f97695c029fef9179d59b03667925d93f8c6608290b838\": rpc error: code = NotFound desc = could not find container \"89db0fcb3c90155b50f97695c029fef9179d59b03667925d93f8c6608290b838\": container with ID starting with 89db0fcb3c90155b50f97695c029fef9179d59b03667925d93f8c6608290b838 not found: ID does not exist" Oct 30 17:34:34.379523 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:34.369605 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2"] Oct 30 17:34:34.382472 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:34.382435 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-15a954ad-openshift-default-86c87b89cc-tbxc2"] Oct 30 17:34:34.545248 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:34:34.545205 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba96a20c_84fa_4a58_916d_0dec80305088.slice/crio-56fe11623ad66e4447b6bbf9b34b78fd3c673e4935d21c6231cd269b80e10165 WatchSource:0}: Error finding container 56fe11623ad66e4447b6bbf9b34b78fd3c673e4935d21c6231cd269b80e10165: Status 404 returned error can't find the container with id 56fe11623ad66e4447b6bbf9b34b78fd3c673e4935d21c6231cd269b80e10165 Oct 30 17:34:34.545514 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:34.545494 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh"] Oct 30 17:34:34.546657 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:34.546639 4965 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:34:35.314113 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:35.314016 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh" event={"ID":"ba96a20c-84fa-4a58-916d-0dec80305088","Type":"ContainerStarted","Data":"806fdb88eddfcf116a50b472912cb565b55c22b84a9927582a36c41e0768d024"} Oct 30 17:34:35.314113 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:35.314084 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh" event={"ID":"ba96a20c-84fa-4a58-916d-0dec80305088","Type":"ContainerStarted","Data":"56fe11623ad66e4447b6bbf9b34b78fd3c673e4935d21c6231cd269b80e10165"} Oct 30 17:34:35.314669 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:35.314117 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh" Oct 30 17:34:35.363148 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:35.363096 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh" podStartSLOduration=1.875037241 podStartE2EDuration="2.363082162s" totalImagesPullingTime="488.044921ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:34:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:34:35.360666636 +0000 UTC m=+1333.412766591" watchObservedRunningTime="2026-10-30 17:34:35.363082162 +0000 UTC m=+1333.415182094" Oct 30 17:34:35.997481 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:34:35.997448 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:34:36.101961 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:36.101926 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52968197-826b-42aa-8627-6287ec75aeed" path="/var/lib/kubelet/pods/52968197-826b-42aa-8627-6287ec75aeed/volumes" Oct 30 17:34:38.930641 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:38.930605 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6"] Oct 30 17:34:38.933997 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:38.933974 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:38.963383 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:38.963351 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6"] Oct 30 17:34:39.006925 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:39.006853 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a628b339-91ea-46d8-9935-fc9174b9e4d2-istio-token\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:39.007129 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:39.006944 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-istio-envoy\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:39.007129 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:39.006993 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-workload-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:39.007129 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:39.007044 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a628b339-91ea-46d8-9935-fc9174b9e4d2-istiod-ca-cert\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:39.007129 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:39.007093 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmlxt\" (UniqueName: \"kubernetes.io/projected/a628b339-91ea-46d8-9935-fc9174b9e4d2-kube-api-access-dmlxt\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:39.007342 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:39.007246 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-credential-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:39.007342 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:39.007270 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-workload-certs\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:39.007440 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:39.007356 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-istio-data\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:39.007538 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:39.007519 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a628b339-91ea-46d8-9935-fc9174b9e4d2-istio-podinfo\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:39.177886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:39.172349 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a628b339-91ea-46d8-9935-fc9174b9e4d2-istio-podinfo\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:39.177886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:39.172397 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a628b339-91ea-46d8-9935-fc9174b9e4d2-istio-token\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:39.177886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:39.172438 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-istio-envoy\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:39.177886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:39.172475 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-workload-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:39.177886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:39.172508 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a628b339-91ea-46d8-9935-fc9174b9e4d2-istiod-ca-cert\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:39.177886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:39.172544 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmlxt\" (UniqueName: \"kubernetes.io/projected/a628b339-91ea-46d8-9935-fc9174b9e4d2-kube-api-access-dmlxt\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:39.177886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:39.172685 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-credential-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:39.177886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:39.172702 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-workload-certs\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:39.177886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:39.172756 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-istio-data\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:39.202068 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:39.201289 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-workload-certs\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:39.202068 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:39.201535 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-credential-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:39.202777 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:39.202750 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-istio-data\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:39.204272 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:39.203992 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a628b339-91ea-46d8-9935-fc9174b9e4d2-istiod-ca-cert\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:39.204417 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:39.204212 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-workload-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:39.215978 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:39.213437 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-istio-envoy\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:39.219441 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:39.219407 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a628b339-91ea-46d8-9935-fc9174b9e4d2-istio-podinfo\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:39.220529 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:39.220498 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmlxt\" (UniqueName: \"kubernetes.io/projected/a628b339-91ea-46d8-9935-fc9174b9e4d2-kube-api-access-dmlxt\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:39.221099 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:39.221081 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a628b339-91ea-46d8-9935-fc9174b9e4d2-istio-token\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:39.247989 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:39.247960 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:39.577374 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:34:39.577343 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda628b339_91ea_46d8_9935_fc9174b9e4d2.slice/crio-3780e05465ee8b08e3441f2c85a70a842ac7039dff370211bfe3e59807423e11 WatchSource:0}: Error finding container 3780e05465ee8b08e3441f2c85a70a842ac7039dff370211bfe3e59807423e11: Status 404 returned error can't find the container with id 3780e05465ee8b08e3441f2c85a70a842ac7039dff370211bfe3e59807423e11 Oct 30 17:34:39.577913 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:39.577895 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6"] Oct 30 17:34:39.580194 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:39.580161 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:34:39.580264 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:39.580224 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:34:39.580264 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:39.580256 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:34:40.360083 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:40.360042 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" event={"ID":"a628b339-91ea-46d8-9935-fc9174b9e4d2","Type":"ContainerStarted","Data":"d2ab40089b0bd25196e93a3fe8fd114e640acd3fbcb87154876db0f62ac9d92f"} Oct 30 17:34:40.360518 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:40.360136 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" event={"ID":"a628b339-91ea-46d8-9935-fc9174b9e4d2","Type":"ContainerStarted","Data":"3780e05465ee8b08e3441f2c85a70a842ac7039dff370211bfe3e59807423e11"} Oct 30 17:34:40.433289 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:40.433233 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" podStartSLOduration=2.433220296 podStartE2EDuration="2.433220296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:34:40.417491727 +0000 UTC m=+1338.469591658" watchObservedRunningTime="2026-10-30 17:34:40.433220296 +0000 UTC m=+1338.485320227" Oct 30 17:34:41.248155 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:41.248115 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:41.250525 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:41.250492 4965 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 17:34:41.250682 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:41.250547 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" podUID="a628b339-91ea-46d8-9935-fc9174b9e4d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 17:34:41.329279 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:41.329252 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh" Oct 30 17:34:42.249293 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:42.249257 4965 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 17:34:42.249703 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:42.249318 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" podUID="a628b339-91ea-46d8-9935-fc9174b9e4d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 17:34:43.248766 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:43.248733 4965 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 17:34:43.248956 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:43.248779 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" podUID="a628b339-91ea-46d8-9935-fc9174b9e4d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 17:34:44.248936 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:44.248895 4965 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 17:34:44.249378 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:44.248949 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" podUID="a628b339-91ea-46d8-9935-fc9174b9e4d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 17:34:45.248833 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:45.248791 4965 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 17:34:45.249043 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:45.248899 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" podUID="a628b339-91ea-46d8-9935-fc9174b9e4d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 17:34:46.249029 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:46.248991 4965 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 17:34:46.249209 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:46.249058 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" podUID="a628b339-91ea-46d8-9935-fc9174b9e4d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 17:34:47.253182 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:47.253150 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:47.253649 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:47.253534 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:47.254174 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:47.254142 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:34:53.221485 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:53.221427 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:34:53.338980 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:53.335214 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 17:34:53.338980 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:53.336053 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 17:34:53.367263 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:53.367220 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:34:53.421941 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:53.420982 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:34:53.461846 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:53.461802 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded" Oct 30 17:34:53.690920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:53.688215 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 17:34:53.882912 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:53.862950 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" podUID="d174a74f-7205-4020-94b0-e30652c9b721" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 30 17:34:53.882912 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:53.876940 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:34:54.274014 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:54.273973 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:34:54.310887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:54.275643 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:34:54.367889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:54.340072 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:34:54.377893 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:54.341466 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:34:54.440893 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:54.439207 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:34:54.496393 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:54.494958 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 17:34:54.682017 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:54.660141 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:34:54.683678 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:54.683651 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:34:54.683927 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:54.683910 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:34:54.727027 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:54.726997 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:34:54.935945 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:54.931551 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" podUID="d174a74f-7205-4020-94b0-e30652c9b721" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:34:54.935945 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:54.932941 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" Oct 30 17:34:54.951154 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:54.937344 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 17:34:54.951154 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:54.938931 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:34:55.056789 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:55.056761 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:34:55.307436 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:55.307338 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:34:55.310003 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:55.308260 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:34:55.339500 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:55.339473 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:34:55.344093 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:55.344069 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" Oct 30 17:34:55.353935 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:55.353843 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:34:55.378556 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:55.378527 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:34:55.392051 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:55.392025 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:34:55.419667 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:34:55.419435 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:35:02.534357 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:02.532894 4965 patch_prober.go:29] interesting pod/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": read tcp 10.132.0.2:36346->10.132.0.98:15021: read: connection reset by peer" start-of-body= Oct 30 17:35:02.534357 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:02.532974 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" podUID="dd97b0c7-6847-48fe-b7bd-06cb3b78935c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": read tcp 10.132.0.2:36346->10.132.0.98:15021: read: connection reset by peer" Oct 30 17:35:02.902117 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:02.902071 4965 generic.go:356] "Generic (PLEG): container finished" podID="dd97b0c7-6847-48fe-b7bd-06cb3b78935c" containerID="da3a8c85785d77a58022c01a7086b6e18addda10b74efc7a3fb607d619edf4f5" exitCode=0 Oct 30 17:35:02.902438 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:02.902235 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" event={"ID":"dd97b0c7-6847-48fe-b7bd-06cb3b78935c","Type":"ContainerDied","Data":"da3a8c85785d77a58022c01a7086b6e18addda10b74efc7a3fb607d619edf4f5"} Oct 30 17:35:02.903197 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:02.903176 4965 scope.go:122] "RemoveContainer" containerID="da3a8c85785d77a58022c01a7086b6e18addda10b74efc7a3fb607d619edf4f5" Oct 30 17:35:02.905161 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:02.905055 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:35:02.905161 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:02.905151 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:35:02.905161 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:02.905206 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:35:03.912561 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:03.912520 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" event={"ID":"dd97b0c7-6847-48fe-b7bd-06cb3b78935c","Type":"ContainerStarted","Data":"1d37a9491773352d540136ec392b812b8528d85e742d33797bd516969c7dd404"} Oct 30 17:35:04.877264 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:04.877229 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:35:04.877616 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:04.877536 4965 patch_prober.go:29] interesting pod/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 17:35:04.877765 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:04.877745 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" podUID="dd97b0c7-6847-48fe-b7bd-06cb3b78935c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 17:35:05.872386 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:05.872343 4965 patch_prober.go:29] interesting pod/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 17:35:05.873024 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:05.872988 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" podUID="dd97b0c7-6847-48fe-b7bd-06cb3b78935c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 17:35:06.010689 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:35:06.010650 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:35:06.022460 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.022427 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k"] Oct 30 17:35:06.026213 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.026189 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:06.050941 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.049797 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-a17354-openshift-default-dockercfg-gqtvv\"" Oct 30 17:35:06.110190 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.110045 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k"] Oct 30 17:35:06.119385 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.119345 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-workload-socket\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:06.119560 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.119404 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b003d430-64cd-4a08-9a96-6b326a534b5e-istio-token\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:06.119560 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.119487 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-istio-envoy\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:06.119697 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.119574 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-workload-certs\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:06.119697 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.119612 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b003d430-64cd-4a08-9a96-6b326a534b5e-istio-podinfo\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:06.119697 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.119690 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b003d430-64cd-4a08-9a96-6b326a534b5e-istiod-ca-cert\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:06.119851 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.119811 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-istio-data\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:06.119949 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.119851 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-credential-socket\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:06.120008 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.119941 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4npr\" (UniqueName: \"kubernetes.io/projected/b003d430-64cd-4a08-9a96-6b326a534b5e-kube-api-access-r4npr\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:06.320949 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.301511 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-workload-certs\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:06.410004 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.409965 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b003d430-64cd-4a08-9a96-6b326a534b5e-istio-podinfo\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:06.410347 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.410326 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b003d430-64cd-4a08-9a96-6b326a534b5e-istiod-ca-cert\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:06.410593 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.410579 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-istio-data\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:06.410704 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.410691 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-credential-socket\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:06.410809 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.410797 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4npr\" (UniqueName: \"kubernetes.io/projected/b003d430-64cd-4a08-9a96-6b326a534b5e-kube-api-access-r4npr\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:06.416893 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.411099 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-workload-socket\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:06.416893 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.411157 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b003d430-64cd-4a08-9a96-6b326a534b5e-istio-token\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:06.416893 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.302342 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-workload-certs\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:06.416893 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.411295 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-istio-envoy\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:06.416893 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.411955 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-workload-socket\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:06.416893 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.412492 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-istio-data\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:06.416893 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.413138 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b003d430-64cd-4a08-9a96-6b326a534b5e-istiod-ca-cert\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:06.416893 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.413379 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-credential-socket\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:06.492888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.490292 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-istio-envoy\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:06.492888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.490819 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b003d430-64cd-4a08-9a96-6b326a534b5e-istio-token\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:06.502944 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.502858 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b003d430-64cd-4a08-9a96-6b326a534b5e-istio-podinfo\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:06.515042 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.514976 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4npr\" (UniqueName: \"kubernetes.io/projected/b003d430-64cd-4a08-9a96-6b326a534b5e-kube-api-access-r4npr\") pod \"e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:06.666933 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.665220 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:06.888306 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.888265 4965 patch_prober.go:29] interesting pod/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 17:35:06.888772 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:06.888334 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" podUID="dd97b0c7-6847-48fe-b7bd-06cb3b78935c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 17:35:07.204110 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:07.204081 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k"] Oct 30 17:35:07.204413 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:35:07.204387 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb003d430_64cd_4a08_9a96_6b326a534b5e.slice/crio-164007abc82072fe9f1f3fb7b2718671783225bb4eca6ce36394e87979df65e1 WatchSource:0}: Error finding container 164007abc82072fe9f1f3fb7b2718671783225bb4eca6ce36394e87979df65e1: Status 404 returned error can't find the container with id 164007abc82072fe9f1f3fb7b2718671783225bb4eca6ce36394e87979df65e1 Oct 30 17:35:07.207181 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:07.207148 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:35:07.207270 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:07.207214 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:35:07.207270 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:07.207245 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:35:07.879749 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:07.879296 4965 patch_prober.go:29] interesting pod/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 17:35:07.879749 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:07.879358 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" podUID="dd97b0c7-6847-48fe-b7bd-06cb3b78935c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 17:35:07.946800 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:07.946761 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" event={"ID":"b003d430-64cd-4a08-9a96-6b326a534b5e","Type":"ContainerStarted","Data":"89c618517e9de02d4828e6dfbb050769a890b36188c3694d68cb8b73e90f163f"} Oct 30 17:35:07.947237 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:07.946827 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" event={"ID":"b003d430-64cd-4a08-9a96-6b326a534b5e","Type":"ContainerStarted","Data":"164007abc82072fe9f1f3fb7b2718671783225bb4eca6ce36394e87979df65e1"} Oct 30 17:35:08.042725 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:08.042666 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" podStartSLOduration=3.042648819 podStartE2EDuration="3.042648819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:35:08.028460793 +0000 UTC m=+1366.080560724" watchObservedRunningTime="2026-10-30 17:35:08.042648819 +0000 UTC m=+1366.094748751" Oct 30 17:35:08.679888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:08.679498 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:08.687108 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:08.687078 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:08.872338 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:08.872297 4965 patch_prober.go:29] interesting pod/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 17:35:08.872645 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:08.872612 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" podUID="dd97b0c7-6847-48fe-b7bd-06cb3b78935c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 17:35:08.954269 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:08.954182 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:08.955225 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:08.955205 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:35:09.825538 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:09.825501 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh"] Oct 30 17:35:09.825813 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:09.825756 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh" podUID="ba96a20c-84fa-4a58-916d-0dec80305088" containerName="maas-api" containerID="cri-o://806fdb88eddfcf116a50b472912cb565b55c22b84a9927582a36c41e0768d024" gracePeriod=30 Oct 30 17:35:09.874196 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:09.874168 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:35:09.875062 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:09.875037 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:35:09.875819 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:09.875797 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:35:09.963811 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:09.963780 4965 generic.go:356] "Generic (PLEG): container finished" podID="ba96a20c-84fa-4a58-916d-0dec80305088" containerID="806fdb88eddfcf116a50b472912cb565b55c22b84a9927582a36c41e0768d024" exitCode=0 Oct 30 17:35:09.964252 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:09.963851 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh" event={"ID":"ba96a20c-84fa-4a58-916d-0dec80305088","Type":"ContainerDied","Data":"806fdb88eddfcf116a50b472912cb565b55c22b84a9927582a36c41e0768d024"} Oct 30 17:35:10.162761 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:10.162734 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh" Oct 30 17:35:10.222663 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:10.222632 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba96a20c-84fa-4a58-916d-0dec80305088-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba96a20c-84fa-4a58-916d-0dec80305088-openshift-service-ca\") pod \"ba96a20c-84fa-4a58-916d-0dec80305088\" (UID: \"ba96a20c-84fa-4a58-916d-0dec80305088\") " Oct 30 17:35:10.222897 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:10.222830 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba96a20c-84fa-4a58-916d-0dec80305088-kube-api-access-v7cgk\" (UniqueName: \"kubernetes.io/projected/ba96a20c-84fa-4a58-916d-0dec80305088-kube-api-access-v7cgk\") pod \"ba96a20c-84fa-4a58-916d-0dec80305088\" (UID: \"ba96a20c-84fa-4a58-916d-0dec80305088\") " Oct 30 17:35:10.223039 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:10.223022 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba96a20c-84fa-4a58-916d-0dec80305088-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba96a20c-84fa-4a58-916d-0dec80305088-maas-api-tls\") pod \"ba96a20c-84fa-4a58-916d-0dec80305088\" (UID: \"ba96a20c-84fa-4a58-916d-0dec80305088\") " Oct 30 17:35:10.223109 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:10.223062 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba96a20c-84fa-4a58-916d-0dec80305088-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba96a20c-84fa-4a58-916d-0dec80305088-metrics-certs\") pod \"ba96a20c-84fa-4a58-916d-0dec80305088\" (UID: \"ba96a20c-84fa-4a58-916d-0dec80305088\") " Oct 30 17:35:10.223174 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:10.223143 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba96a20c-84fa-4a58-916d-0dec80305088-openshift-service-ca" pod "ba96a20c-84fa-4a58-916d-0dec80305088" (UID: "ba96a20c-84fa-4a58-916d-0dec80305088"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:35:10.223751 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:10.223729 4965 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba96a20c-84fa-4a58-916d-0dec80305088-openshift-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:35:10.225748 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:10.225712 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba96a20c-84fa-4a58-916d-0dec80305088-kube-api-access-v7cgk" pod "ba96a20c-84fa-4a58-916d-0dec80305088" (UID: "ba96a20c-84fa-4a58-916d-0dec80305088"). InnerVolumeSpecName "kube-api-access-v7cgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:10.225898 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:10.225792 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba96a20c-84fa-4a58-916d-0dec80305088-maas-api-tls" pod "ba96a20c-84fa-4a58-916d-0dec80305088" (UID: "ba96a20c-84fa-4a58-916d-0dec80305088"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:35:10.225978 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:10.225900 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba96a20c-84fa-4a58-916d-0dec80305088-metrics-certs" pod "ba96a20c-84fa-4a58-916d-0dec80305088" (UID: "ba96a20c-84fa-4a58-916d-0dec80305088"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:35:10.390935 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:10.376199 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba96a20c-84fa-4a58-916d-0dec80305088-maas-api-tls\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:35:10.390935 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:10.376238 4965 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba96a20c-84fa-4a58-916d-0dec80305088-metrics-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:35:10.390935 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:10.376255 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7cgk\" (UniqueName: \"kubernetes.io/projected/ba96a20c-84fa-4a58-916d-0dec80305088-kube-api-access-v7cgk\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:35:10.972072 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:10.972040 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh" Oct 30 17:35:10.972517 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:10.972083 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh" event={"ID":"ba96a20c-84fa-4a58-916d-0dec80305088","Type":"ContainerDied","Data":"56fe11623ad66e4447b6bbf9b34b78fd3c673e4935d21c6231cd269b80e10165"} Oct 30 17:35:10.972517 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:10.972149 4965 scope.go:122] "RemoveContainer" containerID="806fdb88eddfcf116a50b472912cb565b55c22b84a9927582a36c41e0768d024" Oct 30 17:35:11.027574 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:11.027541 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh"] Oct 30 17:35:11.039162 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:11.039137 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-86c97ce8-6d545dd7cc-8s5rh"] Oct 30 17:35:12.100239 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:12.100205 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba96a20c-84fa-4a58-916d-0dec80305088" path="/var/lib/kubelet/pods/ba96a20c-84fa-4a58-916d-0dec80305088/volumes" Oct 30 17:35:13.397090 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:13.397058 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk"] Oct 30 17:35:13.397502 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:13.397243 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" podUID="dd97b0c7-6847-48fe-b7bd-06cb3b78935c" containerName="istio-proxy" containerID="cri-o://1d37a9491773352d540136ec392b812b8528d85e742d33797bd516969c7dd404" gracePeriod=30 Oct 30 17:35:18.724424 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:18.724363 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:35:18.841453 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:18.841411 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-workload-certs\") pod \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " Oct 30 17:35:18.841453 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:18.841487 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istio-data\") pod \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " Oct 30 17:35:18.841453 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:18.841525 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istio-podinfo\") pod \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " Oct 30 17:35:18.841920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:18.841649 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istio-token\" (UniqueName: \"kubernetes.io/projected/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istio-token\") pod \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " Oct 30 17:35:18.841920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:18.841726 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istiod-ca-cert\") pod \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " Oct 30 17:35:18.841920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:18.841761 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-credential-socket\") pod \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " Oct 30 17:35:18.841920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:18.841839 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-workload-socket\") pod \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " Oct 30 17:35:18.842160 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:18.841985 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-kube-api-access-hcm95\" (UniqueName: \"kubernetes.io/projected/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-kube-api-access-hcm95\") pod \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " Oct 30 17:35:18.842780 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:18.842536 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-workload-certs" pod "dd97b0c7-6847-48fe-b7bd-06cb3b78935c" (UID: "dd97b0c7-6847-48fe-b7bd-06cb3b78935c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:35:18.842780 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:18.842647 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-workload-socket" pod "dd97b0c7-6847-48fe-b7bd-06cb3b78935c" (UID: "dd97b0c7-6847-48fe-b7bd-06cb3b78935c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:35:18.842780 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:18.842748 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istiod-ca-cert" pod "dd97b0c7-6847-48fe-b7bd-06cb3b78935c" (UID: "dd97b0c7-6847-48fe-b7bd-06cb3b78935c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:35:18.844319 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:18.843143 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istio-envoy\") pod \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\" (UID: \"dd97b0c7-6847-48fe-b7bd-06cb3b78935c\") " Oct 30 17:35:18.844319 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:18.843988 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:35:18.844319 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:18.844011 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:35:18.844319 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:18.844026 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:35:18.845052 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:18.845023 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-credential-socket" pod "dd97b0c7-6847-48fe-b7bd-06cb3b78935c" (UID: "dd97b0c7-6847-48fe-b7bd-06cb3b78935c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:35:18.845528 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:18.845502 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istio-data" pod "dd97b0c7-6847-48fe-b7bd-06cb3b78935c" (UID: "dd97b0c7-6847-48fe-b7bd-06cb3b78935c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:35:18.845613 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:18.845583 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istio-token" pod "dd97b0c7-6847-48fe-b7bd-06cb3b78935c" (UID: "dd97b0c7-6847-48fe-b7bd-06cb3b78935c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:18.845672 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:18.845616 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-kube-api-access-hcm95" pod "dd97b0c7-6847-48fe-b7bd-06cb3b78935c" (UID: "dd97b0c7-6847-48fe-b7bd-06cb3b78935c"). InnerVolumeSpecName "kube-api-access-hcm95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:18.847383 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:18.847356 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istio-podinfo" pod "dd97b0c7-6847-48fe-b7bd-06cb3b78935c" (UID: "dd97b0c7-6847-48fe-b7bd-06cb3b78935c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:35:18.847539 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:18.847519 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istio-envoy" pod "dd97b0c7-6847-48fe-b7bd-06cb3b78935c" (UID: "dd97b0c7-6847-48fe-b7bd-06cb3b78935c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:35:18.945014 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:18.944984 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:35:18.945014 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:18.945009 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcm95\" (UniqueName: \"kubernetes.io/projected/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-kube-api-access-hcm95\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:35:18.945014 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:18.945020 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:35:18.945234 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:18.945028 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:35:18.945234 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:18.945036 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:35:18.945234 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:18.945044 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd97b0c7-6847-48fe-b7bd-06cb3b78935c-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:35:19.064984 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:19.064842 4965 generic.go:356] "Generic (PLEG): container finished" podID="dd97b0c7-6847-48fe-b7bd-06cb3b78935c" containerID="1d37a9491773352d540136ec392b812b8528d85e742d33797bd516969c7dd404" exitCode=0 Oct 30 17:35:19.064984 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:19.064938 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" event={"ID":"dd97b0c7-6847-48fe-b7bd-06cb3b78935c","Type":"ContainerDied","Data":"1d37a9491773352d540136ec392b812b8528d85e742d33797bd516969c7dd404"} Oct 30 17:35:19.064984 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:19.064963 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" Oct 30 17:35:19.065273 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:19.065042 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk" event={"ID":"dd97b0c7-6847-48fe-b7bd-06cb3b78935c","Type":"ContainerDied","Data":"3b143e33ead3f55b220b9d407f92f2824c44f2c37415ecd998bd6851f3541117"} Oct 30 17:35:19.065273 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:19.065071 4965 scope.go:122] "RemoveContainer" containerID="1d37a9491773352d540136ec392b812b8528d85e742d33797bd516969c7dd404" Oct 30 17:35:19.070198 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:19.070172 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:35:19.078608 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:19.078587 4965 scope.go:122] "RemoveContainer" containerID="da3a8c85785d77a58022c01a7086b6e18addda10b74efc7a3fb607d619edf4f5" Oct 30 17:35:19.088164 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:19.088139 4965 scope.go:122] "RemoveContainer" containerID="1d37a9491773352d540136ec392b812b8528d85e742d33797bd516969c7dd404" Oct 30 17:35:19.088431 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:35:19.088413 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d37a9491773352d540136ec392b812b8528d85e742d33797bd516969c7dd404\": container with ID starting with 1d37a9491773352d540136ec392b812b8528d85e742d33797bd516969c7dd404 not found: ID does not exist" containerID="1d37a9491773352d540136ec392b812b8528d85e742d33797bd516969c7dd404" Oct 30 17:35:19.088475 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:19.088442 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d37a9491773352d540136ec392b812b8528d85e742d33797bd516969c7dd404"} err="failed to get container status \"1d37a9491773352d540136ec392b812b8528d85e742d33797bd516969c7dd404\": rpc error: code = NotFound desc = could not find container \"1d37a9491773352d540136ec392b812b8528d85e742d33797bd516969c7dd404\": container with ID starting with 1d37a9491773352d540136ec392b812b8528d85e742d33797bd516969c7dd404 not found: ID does not exist" Oct 30 17:35:19.088475 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:19.088458 4965 scope.go:122] "RemoveContainer" containerID="da3a8c85785d77a58022c01a7086b6e18addda10b74efc7a3fb607d619edf4f5" Oct 30 17:35:19.088701 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:35:19.088680 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da3a8c85785d77a58022c01a7086b6e18addda10b74efc7a3fb607d619edf4f5\": container with ID starting with da3a8c85785d77a58022c01a7086b6e18addda10b74efc7a3fb607d619edf4f5 not found: ID does not exist" containerID="da3a8c85785d77a58022c01a7086b6e18addda10b74efc7a3fb607d619edf4f5" Oct 30 17:35:19.088760 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:19.088707 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da3a8c85785d77a58022c01a7086b6e18addda10b74efc7a3fb607d619edf4f5"} err="failed to get container status \"da3a8c85785d77a58022c01a7086b6e18addda10b74efc7a3fb607d619edf4f5\": rpc error: code = NotFound desc = could not find container \"da3a8c85785d77a58022c01a7086b6e18addda10b74efc7a3fb607d619edf4f5\": container with ID starting with da3a8c85785d77a58022c01a7086b6e18addda10b74efc7a3fb607d619edf4f5 not found: ID does not exist" Oct 30 17:35:19.126199 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:19.126161 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk"] Oct 30 17:35:19.129711 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:19.129682 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-86c97ce8-openshift-default-668655d98c-7vsjk"] Oct 30 17:35:20.139159 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:20.139124 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd97b0c7-6847-48fe-b7bd-06cb3b78935c" path="/var/lib/kubelet/pods/dd97b0c7-6847-48fe-b7bd-06cb3b78935c/volumes" Oct 30 17:35:23.020203 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.020152 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f"] Oct 30 17:35:23.022159 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.022120 4965 cpu_manager.go:414] "containerMap: removing container" podUID="dd97b0c7-6847-48fe-b7bd-06cb3b78935c" containerName="istio-proxy" Oct 30 17:35:23.022159 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.022161 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd97b0c7-6847-48fe-b7bd-06cb3b78935c" containerName="istio-proxy" Oct 30 17:35:23.022358 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.022200 4965 cpu_manager.go:414] "containerMap: removing container" podUID="dd97b0c7-6847-48fe-b7bd-06cb3b78935c" containerName="istio-proxy" Oct 30 17:35:23.022358 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.022209 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd97b0c7-6847-48fe-b7bd-06cb3b78935c" containerName="istio-proxy" Oct 30 17:35:23.022358 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.022253 4965 cpu_manager.go:414] "containerMap: removing container" podUID="ba96a20c-84fa-4a58-916d-0dec80305088" containerName="maas-api" Oct 30 17:35:23.022358 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.022261 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba96a20c-84fa-4a58-916d-0dec80305088" containerName="maas-api" Oct 30 17:35:23.022563 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.022456 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba96a20c-84fa-4a58-916d-0dec80305088" containerName="maas-api" Oct 30 17:35:23.022563 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.022482 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd97b0c7-6847-48fe-b7bd-06cb3b78935c" containerName="istio-proxy" Oct 30 17:35:23.029700 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.029669 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:23.055788 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.055754 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f"] Oct 30 17:35:23.068484 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.068457 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-81e9336e-openshift-default-dockercfg-4mbp4\"" Oct 30 17:35:23.097399 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.097364 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istio-envoy\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:23.097581 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.097559 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk7lz\" (UniqueName: \"kubernetes.io/projected/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-kube-api-access-sk7lz\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:23.097855 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.097831 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istiod-ca-cert\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:23.098062 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.097972 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istio-podinfo\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:23.098062 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.098016 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-workload-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:23.098200 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.098157 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-credential-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:23.098200 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.098184 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istio-data\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:23.098307 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.098214 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-workload-certs\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:23.098359 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.098338 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istio-token\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:23.200355 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.200321 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk7lz\" (UniqueName: \"kubernetes.io/projected/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-kube-api-access-sk7lz\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:23.200526 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.200438 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istiod-ca-cert\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:23.200526 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.200474 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istio-podinfo\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:23.200526 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.200494 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-workload-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:23.200526 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.200523 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-credential-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:23.200698 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.200548 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istio-data\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:23.200698 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.200588 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-workload-certs\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:23.200698 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.200652 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istio-token\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:23.200892 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.200853 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istio-envoy\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:23.201020 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.201000 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-credential-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:23.201089 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.201054 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-workload-certs\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:23.201142 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.201109 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istio-data\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:23.201283 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.201264 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istiod-ca-cert\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:23.201503 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.201482 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-workload-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:23.203191 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.203165 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istio-envoy\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:23.203436 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.203416 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istio-podinfo\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:23.249453 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.249423 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istio-token\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:23.249787 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.249769 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk7lz\" (UniqueName: \"kubernetes.io/projected/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-kube-api-access-sk7lz\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:23.351280 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:23.351016 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:24.251805 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:24.251721 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a17354-575588998c-57bcf"] Oct 30 17:35:24.253621 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:24.253592 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd97b0c7-6847-48fe-b7bd-06cb3b78935c" containerName="istio-proxy" Oct 30 17:35:24.273636 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:24.273605 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a17354-575588998c-57bcf" Oct 30 17:35:24.280205 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:24.280174 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a17354-575588998c-57bcf"] Oct 30 17:35:24.326705 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:24.317634 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-a17354\"" Oct 30 17:35:24.326705 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:24.317715 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-a17354-dockercfg-lqmlg\"" Oct 30 17:35:24.383903 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:24.338843 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fqh7\" (UniqueName: \"kubernetes.io/projected/404cc4e8-d6c9-4d5b-8abe-c68527da5248-kube-api-access-4fqh7\") pod \"payload-pre-processing-e2e-shared-a-w2-a17354-575588998c-57bcf\" (UID: \"404cc4e8-d6c9-4d5b-8abe-c68527da5248\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a17354-575588998c-57bcf" Oct 30 17:35:24.383903 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:24.382320 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/404cc4e8-d6c9-4d5b-8abe-c68527da5248-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a17354-575588998c-57bcf\" (UID: \"404cc4e8-d6c9-4d5b-8abe-c68527da5248\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a17354-575588998c-57bcf" Oct 30 17:35:24.483686 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:24.483649 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fqh7\" (UniqueName: \"kubernetes.io/projected/404cc4e8-d6c9-4d5b-8abe-c68527da5248-kube-api-access-4fqh7\") pod \"payload-pre-processing-e2e-shared-a-w2-a17354-575588998c-57bcf\" (UID: \"404cc4e8-d6c9-4d5b-8abe-c68527da5248\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a17354-575588998c-57bcf" Oct 30 17:35:24.483859 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:24.483742 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/404cc4e8-d6c9-4d5b-8abe-c68527da5248-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a17354-575588998c-57bcf\" (UID: \"404cc4e8-d6c9-4d5b-8abe-c68527da5248\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a17354-575588998c-57bcf" Oct 30 17:35:24.484556 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:24.484530 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/404cc4e8-d6c9-4d5b-8abe-c68527da5248-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a17354-575588998c-57bcf\" (UID: \"404cc4e8-d6c9-4d5b-8abe-c68527da5248\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a17354-575588998c-57bcf" Oct 30 17:35:24.513664 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:24.513588 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fqh7\" (UniqueName: \"kubernetes.io/projected/404cc4e8-d6c9-4d5b-8abe-c68527da5248-kube-api-access-4fqh7\") pod \"payload-pre-processing-e2e-shared-a-w2-a17354-575588998c-57bcf\" (UID: \"404cc4e8-d6c9-4d5b-8abe-c68527da5248\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a17354-575588998c-57bcf" Oct 30 17:35:24.623292 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:24.623260 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f"] Oct 30 17:35:24.623689 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:35:24.623669 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e4a1789_b3e2_4415_91dd_b9dd1e27cd4d.slice/crio-b35e45ce0319d283ebfd9b1214a7e24dbc7344e449e94637b20602ae0fcb1270 WatchSource:0}: Error finding container b35e45ce0319d283ebfd9b1214a7e24dbc7344e449e94637b20602ae0fcb1270: Status 404 returned error can't find the container with id b35e45ce0319d283ebfd9b1214a7e24dbc7344e449e94637b20602ae0fcb1270 Oct 30 17:35:24.626417 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:24.626382 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:35:24.626496 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:24.626446 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:35:24.626496 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:24.626473 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:35:24.746142 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:24.746104 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a17354-575588998c-57bcf" Oct 30 17:35:25.068610 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.068583 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a17354-575588998c-57bcf"] Oct 30 17:35:25.069157 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:35:25.069133 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404cc4e8_d6c9_4d5b_8abe_c68527da5248.slice/crio-21b1de6fbc5dad5450a029769dafbcd5b5d8e28b1ee6f3d982eb7da2740ba1e5 WatchSource:0}: Error finding container 21b1de6fbc5dad5450a029769dafbcd5b5d8e28b1ee6f3d982eb7da2740ba1e5: Status 404 returned error can't find the container with id 21b1de6fbc5dad5450a029769dafbcd5b5d8e28b1ee6f3d982eb7da2740ba1e5 Oct 30 17:35:25.185814 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.185778 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a17354-575588998c-57bcf" event={"ID":"404cc4e8-d6c9-4d5b-8abe-c68527da5248","Type":"ContainerStarted","Data":"266daea88c68d2ea47ecb788f6fb4c8f5969da95bc710c48e85ead45997ed373"} Oct 30 17:35:25.186012 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.185837 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a17354-575588998c-57bcf" event={"ID":"404cc4e8-d6c9-4d5b-8abe-c68527da5248","Type":"ContainerStarted","Data":"21b1de6fbc5dad5450a029769dafbcd5b5d8e28b1ee6f3d982eb7da2740ba1e5"} Oct 30 17:35:25.186012 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.185876 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a17354-575588998c-57bcf" Oct 30 17:35:25.187639 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.187607 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" event={"ID":"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d","Type":"ContainerStarted","Data":"778ff37a1424c5358b1ceaa72ddecac6f3bb1906ce75db337be7471a380bc0d0"} Oct 30 17:35:25.187776 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.187651 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" event={"ID":"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d","Type":"ContainerStarted","Data":"b35e45ce0319d283ebfd9b1214a7e24dbc7344e449e94637b20602ae0fcb1270"} Oct 30 17:35:25.234197 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.234139 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a17354-575588998c-57bcf" podStartSLOduration=2.234124147 podStartE2EDuration="2.234124147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:35:25.230989233 +0000 UTC m=+1383.283089185" watchObservedRunningTime="2026-10-30 17:35:25.234124147 +0000 UTC m=+1383.286224078" Oct 30 17:35:25.286724 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.286666 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" podStartSLOduration=3.286653529 podStartE2EDuration="3.286653529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:35:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:35:25.285231235 +0000 UTC m=+1383.337331164" watchObservedRunningTime="2026-10-30 17:35:25.286653529 +0000 UTC m=+1383.338753459" Oct 30 17:35:25.351935 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.351902 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:25.358330 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.358299 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:25.514995 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.514958 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2"] Oct 30 17:35:25.519830 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.519807 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:25.550105 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.550069 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2"] Oct 30 17:35:25.551388 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.551336 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-c31c73-openshift-default-dockercfg-qnblk\"" Oct 30 17:35:25.601769 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.601729 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-workload-socket\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:25.601982 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.601843 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6497147f-bad2-434c-a6e4-45b247b75fc2-istiod-ca-cert\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:25.601982 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.601951 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-credential-socket\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:25.602062 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.601999 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-istio-data\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:25.602115 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.602099 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6497147f-bad2-434c-a6e4-45b247b75fc2-istio-podinfo\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:25.602194 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.602180 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-istio-envoy\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:25.602281 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.602268 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k698f\" (UniqueName: \"kubernetes.io/projected/6497147f-bad2-434c-a6e4-45b247b75fc2-kube-api-access-k698f\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:25.602338 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.602325 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-workload-certs\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:25.602421 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.602405 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6497147f-bad2-434c-a6e4-45b247b75fc2-istio-token\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:25.703798 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.703765 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6497147f-bad2-434c-a6e4-45b247b75fc2-istio-podinfo\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:25.704022 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.703931 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-istio-envoy\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:25.704022 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.703998 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k698f\" (UniqueName: \"kubernetes.io/projected/6497147f-bad2-434c-a6e4-45b247b75fc2-kube-api-access-k698f\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:25.704108 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.704059 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-workload-certs\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:25.704151 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.704123 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6497147f-bad2-434c-a6e4-45b247b75fc2-istio-token\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:25.704251 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.704234 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-workload-socket\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:25.704312 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.704296 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6497147f-bad2-434c-a6e4-45b247b75fc2-istiod-ca-cert\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:25.704371 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.704336 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-credential-socket\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:25.704423 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.704382 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-istio-data\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:25.704525 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.704466 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-workload-certs\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:25.704608 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.704583 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-workload-socket\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:25.705065 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.704855 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-credential-socket\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:25.705065 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.704993 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-istio-data\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:25.705191 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.705088 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6497147f-bad2-434c-a6e4-45b247b75fc2-istiod-ca-cert\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:25.706380 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.706356 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-istio-envoy\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:25.706778 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.706756 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6497147f-bad2-434c-a6e4-45b247b75fc2-istio-podinfo\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:25.739634 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.739609 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6497147f-bad2-434c-a6e4-45b247b75fc2-istio-token\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:25.739764 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.739686 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k698f\" (UniqueName: \"kubernetes.io/projected/6497147f-bad2-434c-a6e4-45b247b75fc2-kube-api-access-k698f\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:25.839634 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:25.838220 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:26.223978 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:26.215753 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:26.223978 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:26.220614 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:35:26.518228 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:26.518194 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2"] Oct 30 17:35:26.518664 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:35:26.518331 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6497147f_bad2_434c_a6e4_45b247b75fc2.slice/crio-f806d53241173bc857f8435ee8bcce81d16e8b6c6fcb62ffcd9a4c411bed7202 WatchSource:0}: Error finding container f806d53241173bc857f8435ee8bcce81d16e8b6c6fcb62ffcd9a4c411bed7202: Status 404 returned error can't find the container with id f806d53241173bc857f8435ee8bcce81d16e8b6c6fcb62ffcd9a4c411bed7202 Oct 30 17:35:26.521217 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:26.521183 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:35:26.521306 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:26.521248 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:35:26.521306 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:26.521283 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:35:27.220295 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:27.220253 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" event={"ID":"6497147f-bad2-434c-a6e4-45b247b75fc2","Type":"ContainerStarted","Data":"1ec22adeee1cac2f41e7428a64b322f23700d26c42777a02438683ab45754b0f"} Oct 30 17:35:27.220472 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:27.220317 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" event={"ID":"6497147f-bad2-434c-a6e4-45b247b75fc2","Type":"ContainerStarted","Data":"f806d53241173bc857f8435ee8bcce81d16e8b6c6fcb62ffcd9a4c411bed7202"} Oct 30 17:35:27.271195 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:27.271134 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" podStartSLOduration=2.2711203380000002 podStartE2EDuration="2.271120338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:35:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:35:27.265846873 +0000 UTC m=+1385.317946835" watchObservedRunningTime="2026-10-30 17:35:27.271120338 +0000 UTC m=+1385.323220268" Oct 30 17:35:27.847554 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:27.843226 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:27.857054 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:27.851911 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:28.227292 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:28.227210 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:28.228135 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:28.228113 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:35:35.997515 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:35:35.997424 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:35:36.222571 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:36.222544 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a17354-575588998c-57bcf" Oct 30 17:35:40.508781 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:40.508736 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-phmnv"] Oct 30 17:35:40.513579 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:40.513552 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-phmnv" Oct 30 17:35:40.562165 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:40.562124 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-81e9336e\"" Oct 30 17:35:40.563440 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:40.563418 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-81e9336e-dockercfg-h9b4n\"" Oct 30 17:35:40.570157 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:40.570121 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-phmnv"] Oct 30 17:35:40.619149 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:40.619104 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kftgh\" (UniqueName: \"kubernetes.io/projected/ab8cfab6-dffc-4922-9106-12ca7f613b5b-kube-api-access-kftgh\") pod \"payload-processing-e2e-mt-81e9336e-5f87896b8-phmnv\" (UID: \"ab8cfab6-dffc-4922-9106-12ca7f613b5b\") " pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-phmnv" Oct 30 17:35:40.619350 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:40.619232 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8cfab6-dffc-4922-9106-12ca7f613b5b-plugins-config-volume\") pod \"payload-processing-e2e-mt-81e9336e-5f87896b8-phmnv\" (UID: \"ab8cfab6-dffc-4922-9106-12ca7f613b5b\") " pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-phmnv" Oct 30 17:35:40.797982 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:40.779851 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kftgh\" (UniqueName: \"kubernetes.io/projected/ab8cfab6-dffc-4922-9106-12ca7f613b5b-kube-api-access-kftgh\") pod \"payload-processing-e2e-mt-81e9336e-5f87896b8-phmnv\" (UID: \"ab8cfab6-dffc-4922-9106-12ca7f613b5b\") " pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-phmnv" Oct 30 17:35:40.797982 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:40.779995 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8cfab6-dffc-4922-9106-12ca7f613b5b-plugins-config-volume\") pod \"payload-processing-e2e-mt-81e9336e-5f87896b8-phmnv\" (UID: \"ab8cfab6-dffc-4922-9106-12ca7f613b5b\") " pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-phmnv" Oct 30 17:35:40.797982 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:40.781371 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8cfab6-dffc-4922-9106-12ca7f613b5b-plugins-config-volume\") pod \"payload-processing-e2e-mt-81e9336e-5f87896b8-phmnv\" (UID: \"ab8cfab6-dffc-4922-9106-12ca7f613b5b\") " pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-phmnv" Oct 30 17:35:40.890137 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:40.885656 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kftgh\" (UniqueName: \"kubernetes.io/projected/ab8cfab6-dffc-4922-9106-12ca7f613b5b-kube-api-access-kftgh\") pod \"payload-processing-e2e-mt-81e9336e-5f87896b8-phmnv\" (UID: \"ab8cfab6-dffc-4922-9106-12ca7f613b5b\") " pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-phmnv" Oct 30 17:35:40.901325 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:40.901252 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-phmnv" Oct 30 17:35:41.312836 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:41.312803 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-phmnv"] Oct 30 17:35:41.384709 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:41.384670 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-phmnv" event={"ID":"ab8cfab6-dffc-4922-9106-12ca7f613b5b","Type":"ContainerStarted","Data":"47c3f3542a4aade1c1096ba1198db84e4cb8fc084dafce45282505e8302802f7"} Oct 30 17:35:42.381242 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:42.381210 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-c31c73-7c486cc84-tggjn"] Oct 30 17:35:42.385801 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:42.385774 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c31c73-7c486cc84-tggjn" Oct 30 17:35:42.396266 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:42.396240 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-phmnv" event={"ID":"ab8cfab6-dffc-4922-9106-12ca7f613b5b","Type":"ContainerStarted","Data":"848305e4e194e9b95c117065ac7a93df5f5cc7c4b936c1787c292e233d190cab"} Oct 30 17:35:42.396388 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:42.396356 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-phmnv" Oct 30 17:35:42.402966 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:42.402943 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-c31c73\"" Oct 30 17:35:42.420123 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:42.420095 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-c31c73-7c486cc84-tggjn"] Oct 30 17:35:42.461732 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:42.461698 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9lkg\" (UniqueName: \"kubernetes.io/projected/7357d692-af47-49e1-a236-dd584604557e-kube-api-access-t9lkg\") pod \"payload-processing-e2e-shared-a-w0-c31c73-7c486cc84-tggjn\" (UID: \"7357d692-af47-49e1-a236-dd584604557e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c31c73-7c486cc84-tggjn" Oct 30 17:35:42.461905 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:42.461792 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7357d692-af47-49e1-a236-dd584604557e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-c31c73-7c486cc84-tggjn\" (UID: \"7357d692-af47-49e1-a236-dd584604557e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c31c73-7c486cc84-tggjn" Oct 30 17:35:42.462778 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:42.462753 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c31c73-b54b865dc-568gf"] Oct 30 17:35:42.467871 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:42.467842 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c31c73-b54b865dc-568gf" Oct 30 17:35:42.485367 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:42.485345 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-c31c73-dockercfg-t7l8z\"" Oct 30 17:35:42.486313 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:42.486293 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c31c73-b54b865dc-568gf"] Oct 30 17:35:42.498899 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:42.498824 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-phmnv" podStartSLOduration=2.498800019 podStartE2EDuration="2.498800019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:35:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:35:42.491931009 +0000 UTC m=+1400.544030948" watchObservedRunningTime="2026-10-30 17:35:42.498800019 +0000 UTC m=+1400.550899951" Oct 30 17:35:42.711007 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:42.696434 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bd96d50-e3b8-419c-9d18-20b88a540412-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-c31c73-b54b865dc-568gf\" (UID: \"1bd96d50-e3b8-419c-9d18-20b88a540412\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c31c73-b54b865dc-568gf" Oct 30 17:35:42.711007 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:42.696518 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9lkg\" (UniqueName: \"kubernetes.io/projected/7357d692-af47-49e1-a236-dd584604557e-kube-api-access-t9lkg\") pod \"payload-processing-e2e-shared-a-w0-c31c73-7c486cc84-tggjn\" (UID: \"7357d692-af47-49e1-a236-dd584604557e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c31c73-7c486cc84-tggjn" Oct 30 17:35:42.711007 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:42.696602 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7357d692-af47-49e1-a236-dd584604557e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-c31c73-7c486cc84-tggjn\" (UID: \"7357d692-af47-49e1-a236-dd584604557e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c31c73-7c486cc84-tggjn" Oct 30 17:35:42.711007 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:42.696822 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzwzh\" (UniqueName: \"kubernetes.io/projected/1bd96d50-e3b8-419c-9d18-20b88a540412-kube-api-access-lzwzh\") pod \"payload-pre-processing-e2e-shared-a-w0-c31c73-b54b865dc-568gf\" (UID: \"1bd96d50-e3b8-419c-9d18-20b88a540412\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c31c73-b54b865dc-568gf" Oct 30 17:35:42.711007 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:42.698432 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7357d692-af47-49e1-a236-dd584604557e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-c31c73-7c486cc84-tggjn\" (UID: \"7357d692-af47-49e1-a236-dd584604557e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c31c73-7c486cc84-tggjn" Oct 30 17:35:42.746710 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:42.746152 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9lkg\" (UniqueName: \"kubernetes.io/projected/7357d692-af47-49e1-a236-dd584604557e-kube-api-access-t9lkg\") pod \"payload-processing-e2e-shared-a-w0-c31c73-7c486cc84-tggjn\" (UID: \"7357d692-af47-49e1-a236-dd584604557e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c31c73-7c486cc84-tggjn" Oct 30 17:35:42.813887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:42.800317 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bd96d50-e3b8-419c-9d18-20b88a540412-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-c31c73-b54b865dc-568gf\" (UID: \"1bd96d50-e3b8-419c-9d18-20b88a540412\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c31c73-b54b865dc-568gf" Oct 30 17:35:42.813887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:42.800716 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzwzh\" (UniqueName: \"kubernetes.io/projected/1bd96d50-e3b8-419c-9d18-20b88a540412-kube-api-access-lzwzh\") pod \"payload-pre-processing-e2e-shared-a-w0-c31c73-b54b865dc-568gf\" (UID: \"1bd96d50-e3b8-419c-9d18-20b88a540412\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c31c73-b54b865dc-568gf" Oct 30 17:35:42.813887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:42.802700 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bd96d50-e3b8-419c-9d18-20b88a540412-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-c31c73-b54b865dc-568gf\" (UID: \"1bd96d50-e3b8-419c-9d18-20b88a540412\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c31c73-b54b865dc-568gf" Oct 30 17:35:42.881883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:42.877666 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzwzh\" (UniqueName: \"kubernetes.io/projected/1bd96d50-e3b8-419c-9d18-20b88a540412-kube-api-access-lzwzh\") pod \"payload-pre-processing-e2e-shared-a-w0-c31c73-b54b865dc-568gf\" (UID: \"1bd96d50-e3b8-419c-9d18-20b88a540412\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c31c73-b54b865dc-568gf" Oct 30 17:35:43.002938 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:43.002158 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c31c73-7c486cc84-tggjn" Oct 30 17:35:43.085466 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:43.083411 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c31c73-b54b865dc-568gf" Oct 30 17:35:43.528329 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:35:43.528289 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bd96d50_e3b8_419c_9d18_20b88a540412.slice/crio-79254178d24cdd9c5345e6a723056e4379309e731518676231a5bc5f8fc49954 WatchSource:0}: Error finding container 79254178d24cdd9c5345e6a723056e4379309e731518676231a5bc5f8fc49954: Status 404 returned error can't find the container with id 79254178d24cdd9c5345e6a723056e4379309e731518676231a5bc5f8fc49954 Oct 30 17:35:43.530052 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:43.529986 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c31c73-b54b865dc-568gf"] Oct 30 17:35:43.538925 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:43.538903 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-c31c73-7c486cc84-tggjn"] Oct 30 17:35:43.541202 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:35:43.541180 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7357d692_af47_49e1_a236_dd584604557e.slice/crio-41cb2559cea2372f97201d174075a23bf805a57b0243c59ca8add4a5b1a4cf3e WatchSource:0}: Error finding container 41cb2559cea2372f97201d174075a23bf805a57b0243c59ca8add4a5b1a4cf3e: Status 404 returned error can't find the container with id 41cb2559cea2372f97201d174075a23bf805a57b0243c59ca8add4a5b1a4cf3e Oct 30 17:35:44.411132 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:44.411095 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c31c73-7c486cc84-tggjn" event={"ID":"7357d692-af47-49e1-a236-dd584604557e","Type":"ContainerStarted","Data":"6e297a1d7e19a64de4b45a043bc2fd9d37812546e9a66b28099ebe21df9197ec"} Oct 30 17:35:44.411322 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:44.411163 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c31c73-7c486cc84-tggjn" Oct 30 17:35:44.411322 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:44.411185 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c31c73-7c486cc84-tggjn" event={"ID":"7357d692-af47-49e1-a236-dd584604557e","Type":"ContainerStarted","Data":"41cb2559cea2372f97201d174075a23bf805a57b0243c59ca8add4a5b1a4cf3e"} Oct 30 17:35:44.412846 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:44.412817 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c31c73-b54b865dc-568gf" event={"ID":"1bd96d50-e3b8-419c-9d18-20b88a540412","Type":"ContainerStarted","Data":"51663491535a5f6ab3ec3aedbcd4fa0607c2594b12d5ad2092057346761a3f77"} Oct 30 17:35:44.413007 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:44.412891 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c31c73-b54b865dc-568gf" event={"ID":"1bd96d50-e3b8-419c-9d18-20b88a540412","Type":"ContainerStarted","Data":"79254178d24cdd9c5345e6a723056e4379309e731518676231a5bc5f8fc49954"} Oct 30 17:35:44.413007 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:44.412912 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c31c73-b54b865dc-568gf" Oct 30 17:35:44.472921 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:44.472854 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c31c73-7c486cc84-tggjn" podStartSLOduration=2.472840013 podStartE2EDuration="2.472840013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:35:44.471395587 +0000 UTC m=+1402.523495518" watchObservedRunningTime="2026-10-30 17:35:44.472840013 +0000 UTC m=+1402.524939943" Oct 30 17:35:44.534940 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:44.534890 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c31c73-b54b865dc-568gf" podStartSLOduration=2.534876827 podStartE2EDuration="2.534876827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:35:44.526249871 +0000 UTC m=+1402.578349842" watchObservedRunningTime="2026-10-30 17:35:44.534876827 +0000 UTC m=+1402.586976751" Oct 30 17:35:48.220007 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:48.209924 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:35:53.452913 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:53.442047 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-phmnv" Oct 30 17:35:55.423507 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:55.423474 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c31c73-b54b865dc-568gf" Oct 30 17:35:55.423929 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:55.423550 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c31c73-7c486cc84-tggjn" Oct 30 17:35:59.812449 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:59.812417 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7bb2e69d-54765d4d8c-kvxq7"] Oct 30 17:35:59.818575 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:59.818545 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7bb2e69d-54765d4d8c-kvxq7" Oct 30 17:35:59.845163 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:59.845062 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5dmx\" (UniqueName: \"kubernetes.io/projected/a48121a5-ca41-4b77-a554-a08b069c97ea-kube-api-access-c5dmx\") pod \"payload-pre-processing-e2e-mt-7bb2e69d-54765d4d8c-kvxq7\" (UID: \"a48121a5-ca41-4b77-a554-a08b069c97ea\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7bb2e69d-54765d4d8c-kvxq7" Oct 30 17:35:59.845351 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:59.845325 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a48121a5-ca41-4b77-a554-a08b069c97ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7bb2e69d-54765d4d8c-kvxq7\" (UID: \"a48121a5-ca41-4b77-a554-a08b069c97ea\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7bb2e69d-54765d4d8c-kvxq7" Oct 30 17:35:59.848264 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:59.848235 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7bb2e69d-54765d4d8c-kvxq7"] Oct 30 17:35:59.891899 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:59.876531 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7bb2e69d\"" Oct 30 17:35:59.947890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:59.941464 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7bb2e69d-dockercfg-s7vms\"" Oct 30 17:35:59.976904 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:59.961364 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5dmx\" (UniqueName: \"kubernetes.io/projected/a48121a5-ca41-4b77-a554-a08b069c97ea-kube-api-access-c5dmx\") pod \"payload-pre-processing-e2e-mt-7bb2e69d-54765d4d8c-kvxq7\" (UID: \"a48121a5-ca41-4b77-a554-a08b069c97ea\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7bb2e69d-54765d4d8c-kvxq7" Oct 30 17:36:00.021951 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:35:59.997485 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a48121a5-ca41-4b77-a554-a08b069c97ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7bb2e69d-54765d4d8c-kvxq7\" (UID: \"a48121a5-ca41-4b77-a554-a08b069c97ea\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7bb2e69d-54765d4d8c-kvxq7" Oct 30 17:36:00.032582 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:00.022763 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a48121a5-ca41-4b77-a554-a08b069c97ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7bb2e69d-54765d4d8c-kvxq7\" (UID: \"a48121a5-ca41-4b77-a554-a08b069c97ea\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7bb2e69d-54765d4d8c-kvxq7" Oct 30 17:36:00.196889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:00.196749 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5dmx\" (UniqueName: \"kubernetes.io/projected/a48121a5-ca41-4b77-a554-a08b069c97ea-kube-api-access-c5dmx\") pod \"payload-pre-processing-e2e-mt-7bb2e69d-54765d4d8c-kvxq7\" (UID: \"a48121a5-ca41-4b77-a554-a08b069c97ea\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7bb2e69d-54765d4d8c-kvxq7" Oct 30 17:36:00.205902 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:00.205650 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7bb2e69d-54765d4d8c-kvxq7" Oct 30 17:36:00.626067 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:00.626038 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7bb2e69d-54765d4d8c-kvxq7"] Oct 30 17:36:00.627222 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:36:00.627182 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda48121a5_ca41_4b77_a554_a08b069c97ea.slice/crio-34f31ece8f16dadc9380f8cf436959daf6b6d22465b7cc98e372cf71f65c1122 WatchSource:0}: Error finding container 34f31ece8f16dadc9380f8cf436959daf6b6d22465b7cc98e372cf71f65c1122: Status 404 returned error can't find the container with id 34f31ece8f16dadc9380f8cf436959daf6b6d22465b7cc98e372cf71f65c1122 Oct 30 17:36:00.782384 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:00.782335 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7bb2e69d-54765d4d8c-kvxq7" event={"ID":"a48121a5-ca41-4b77-a554-a08b069c97ea","Type":"ContainerStarted","Data":"662777fe077a2253d601ada61bbfcd106caf68dda79d8b2888307560dfad715d"} Oct 30 17:36:00.783347 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:00.782402 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7bb2e69d-54765d4d8c-kvxq7" event={"ID":"a48121a5-ca41-4b77-a554-a08b069c97ea","Type":"ContainerStarted","Data":"34f31ece8f16dadc9380f8cf436959daf6b6d22465b7cc98e372cf71f65c1122"} Oct 30 17:36:00.783347 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:00.782432 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7bb2e69d-54765d4d8c-kvxq7" Oct 30 17:36:00.836222 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:00.836167 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7bb2e69d-54765d4d8c-kvxq7" podStartSLOduration=1.8361527949999998 podStartE2EDuration="1.836152795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:00.832101162 +0000 UTC m=+1418.884201125" watchObservedRunningTime="2026-10-30 17:36:00.836152795 +0000 UTC m=+1418.888252725" Oct 30 17:36:01.794543 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:01.794501 4965 generic.go:356] "Generic (PLEG): container finished" podID="b003d430-64cd-4a08-9a96-6b326a534b5e" containerID="89c618517e9de02d4828e6dfbb050769a890b36188c3694d68cb8b73e90f163f" exitCode=0 Oct 30 17:36:01.794745 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:01.794581 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" event={"ID":"b003d430-64cd-4a08-9a96-6b326a534b5e","Type":"ContainerDied","Data":"89c618517e9de02d4828e6dfbb050769a890b36188c3694d68cb8b73e90f163f"} Oct 30 17:36:01.795310 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:01.795293 4965 scope.go:122] "RemoveContainer" containerID="89c618517e9de02d4828e6dfbb050769a890b36188c3694d68cb8b73e90f163f" Oct 30 17:36:01.796774 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:01.796741 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:36:01.796903 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:01.796822 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:36:01.796903 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:01.796888 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:36:03.045886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:03.035582 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" event={"ID":"b003d430-64cd-4a08-9a96-6b326a534b5e","Type":"ContainerStarted","Data":"efb393c33e8a75e090b7e950d53b95bd27c15cf059e5ca90be6ad2119efd36f6"} Oct 30 17:36:03.669149 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:03.669118 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:36:03.669537 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:03.669513 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 17:36:03.669716 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:03.669689 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" podUID="b003d430-64cd-4a08-9a96-6b326a534b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 17:36:04.678888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:04.669230 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 17:36:04.678888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:04.669291 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" podUID="b003d430-64cd-4a08-9a96-6b326a534b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 17:36:05.681309 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:05.681268 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 17:36:05.681969 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:05.681936 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" podUID="b003d430-64cd-4a08-9a96-6b326a534b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 17:36:06.010096 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:36:06.010003 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:36:06.680890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:06.670239 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 17:36:06.680890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:06.670298 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" podUID="b003d430-64cd-4a08-9a96-6b326a534b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 17:36:07.683066 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:07.683036 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 17:36:07.683492 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:07.683094 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" podUID="b003d430-64cd-4a08-9a96-6b326a534b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 17:36:08.672502 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:08.672472 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:36:08.673055 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:08.673030 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:36:08.673679 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:08.673659 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 17:36:08.740498 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:08.740465 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016f1d-5d984ddf9-sjb6b"] Oct 30 17:36:08.744007 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:08.743986 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016f1d-5d984ddf9-sjb6b" Oct 30 17:36:08.762966 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:08.762938 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-016f1d\"" Oct 30 17:36:08.763253 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:08.763229 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-016f1d-dockercfg-h78k6\"" Oct 30 17:36:08.766021 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:08.765997 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016f1d-5d984ddf9-sjb6b"] Oct 30 17:36:08.798486 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:08.798457 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-016f1d-78b77d5dc4-qxvr6"] Oct 30 17:36:08.802025 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:08.801999 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016f1d-78b77d5dc4-qxvr6" Oct 30 17:36:08.844829 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:08.844795 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-016f1d-78b77d5dc4-qxvr6"] Oct 30 17:36:08.899281 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:08.899237 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84fbd1da-e017-490a-8db0-1c9555ceb3b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-016f1d-5d984ddf9-sjb6b\" (UID: \"84fbd1da-e017-490a-8db0-1c9555ceb3b4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016f1d-5d984ddf9-sjb6b" Oct 30 17:36:08.899777 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:08.899723 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xksz4\" (UniqueName: \"kubernetes.io/projected/84fbd1da-e017-490a-8db0-1c9555ceb3b4-kube-api-access-xksz4\") pod \"payload-pre-processing-e2e-shared-b-w2-016f1d-5d984ddf9-sjb6b\" (UID: \"84fbd1da-e017-490a-8db0-1c9555ceb3b4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016f1d-5d984ddf9-sjb6b" Oct 30 17:36:08.900773 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:08.899927 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz685\" (UniqueName: \"kubernetes.io/projected/17d99e83-45e9-446a-9525-ee96fe9ff19a-kube-api-access-qz685\") pod \"payload-processing-e2e-shared-b-w2-016f1d-78b77d5dc4-qxvr6\" (UID: \"17d99e83-45e9-446a-9525-ee96fe9ff19a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016f1d-78b77d5dc4-qxvr6" Oct 30 17:36:08.900773 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:08.899975 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17d99e83-45e9-446a-9525-ee96fe9ff19a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-016f1d-78b77d5dc4-qxvr6\" (UID: \"17d99e83-45e9-446a-9525-ee96fe9ff19a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016f1d-78b77d5dc4-qxvr6" Oct 30 17:36:08.919915 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:08.919893 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m"] Oct 30 17:36:08.923237 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:08.923192 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:08.997889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:08.968439 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-8d1062-openshift-default-dockercfg-xm479\"" Oct 30 17:36:09.019889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.003944 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84fbd1da-e017-490a-8db0-1c9555ceb3b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-016f1d-5d984ddf9-sjb6b\" (UID: \"84fbd1da-e017-490a-8db0-1c9555ceb3b4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016f1d-5d984ddf9-sjb6b" Oct 30 17:36:09.020310 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.020282 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-workload-socket\") pod \"e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:09.020661 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.020639 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-credential-socket\") pod \"e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:09.020904 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.020878 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84fbd1da-e017-490a-8db0-1c9555ceb3b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-016f1d-5d984ddf9-sjb6b\" (UID: \"84fbd1da-e017-490a-8db0-1c9555ceb3b4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016f1d-5d984ddf9-sjb6b" Oct 30 17:36:09.021095 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.021074 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istio-envoy\") pod \"e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:09.021286 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.021269 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istio-podinfo\") pod \"e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:09.021457 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.021439 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istio-data\") pod \"e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:09.078803 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.073041 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xksz4\" (UniqueName: \"kubernetes.io/projected/84fbd1da-e017-490a-8db0-1c9555ceb3b4-kube-api-access-xksz4\") pod \"payload-pre-processing-e2e-shared-b-w2-016f1d-5d984ddf9-sjb6b\" (UID: \"84fbd1da-e017-490a-8db0-1c9555ceb3b4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016f1d-5d984ddf9-sjb6b" Oct 30 17:36:09.078803 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.073098 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-422h5\" (UniqueName: \"kubernetes.io/projected/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-kube-api-access-422h5\") pod \"e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:09.078803 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.073175 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-workload-certs\") pod \"e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:09.078803 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.073220 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istiod-ca-cert\") pod \"e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:09.078803 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.073252 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istio-token\") pod \"e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:09.078803 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.073326 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz685\" (UniqueName: \"kubernetes.io/projected/17d99e83-45e9-446a-9525-ee96fe9ff19a-kube-api-access-qz685\") pod \"payload-processing-e2e-shared-b-w2-016f1d-78b77d5dc4-qxvr6\" (UID: \"17d99e83-45e9-446a-9525-ee96fe9ff19a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016f1d-78b77d5dc4-qxvr6" Oct 30 17:36:09.078803 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.073368 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17d99e83-45e9-446a-9525-ee96fe9ff19a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-016f1d-78b77d5dc4-qxvr6\" (UID: \"17d99e83-45e9-446a-9525-ee96fe9ff19a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016f1d-78b77d5dc4-qxvr6" Oct 30 17:36:09.078803 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.074500 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17d99e83-45e9-446a-9525-ee96fe9ff19a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-016f1d-78b77d5dc4-qxvr6\" (UID: \"17d99e83-45e9-446a-9525-ee96fe9ff19a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016f1d-78b77d5dc4-qxvr6" Oct 30 17:36:09.163991 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.157742 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz685\" (UniqueName: \"kubernetes.io/projected/17d99e83-45e9-446a-9525-ee96fe9ff19a-kube-api-access-qz685\") pod \"payload-processing-e2e-shared-b-w2-016f1d-78b77d5dc4-qxvr6\" (UID: \"17d99e83-45e9-446a-9525-ee96fe9ff19a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016f1d-78b77d5dc4-qxvr6" Oct 30 17:36:09.180966 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.180618 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-workload-socket\") pod \"e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:09.180966 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.180717 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-credential-socket\") pod \"e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:09.180966 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.180776 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istio-envoy\") pod \"e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:09.183824 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.181269 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istio-podinfo\") pod \"e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:09.183824 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.181381 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istio-data\") pod \"e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:09.183824 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.181471 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-422h5\" (UniqueName: \"kubernetes.io/projected/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-kube-api-access-422h5\") pod \"e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:09.183824 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.181568 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-workload-certs\") pod \"e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:09.183824 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.181620 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istiod-ca-cert\") pod \"e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:09.183824 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.181653 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istio-token\") pod \"e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:09.183824 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.182745 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-workload-socket\") pod \"e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:09.183824 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.183097 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-credential-socket\") pod \"e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:09.183824 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.183360 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-workload-certs\") pod \"e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:09.186011 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.184705 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istiod-ca-cert\") pod \"e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:09.186428 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.186406 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istio-data\") pod \"e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:09.191204 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.191160 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istio-podinfo\") pod \"e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:09.196909 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.196743 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istio-envoy\") pod \"e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:09.206969 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.197813 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xksz4\" (UniqueName: \"kubernetes.io/projected/84fbd1da-e017-490a-8db0-1c9555ceb3b4-kube-api-access-xksz4\") pod \"payload-pre-processing-e2e-shared-b-w2-016f1d-5d984ddf9-sjb6b\" (UID: \"84fbd1da-e017-490a-8db0-1c9555ceb3b4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016f1d-5d984ddf9-sjb6b" Oct 30 17:36:09.229104 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.229021 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-422h5\" (UniqueName: \"kubernetes.io/projected/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-kube-api-access-422h5\") pod \"e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:09.232846 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.232787 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istio-token\") pod \"e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:09.235581 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.235533 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m"] Oct 30 17:36:09.236293 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.236270 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:09.378713 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.364515 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016f1d-5d984ddf9-sjb6b" Oct 30 17:36:09.425572 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.423161 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016f1d-78b77d5dc4-qxvr6" Oct 30 17:36:09.760952 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.760855 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m"] Oct 30 17:36:09.774641 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:36:09.774612 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82a26e93_a3d6_4ea9_ae39_edda4cd2fc0e.slice/crio-31dd8b0c4bfd1b74a298b4ae4750ffeb653b50bc59da82ab548a9244e3c34b86 WatchSource:0}: Error finding container 31dd8b0c4bfd1b74a298b4ae4750ffeb653b50bc59da82ab548a9244e3c34b86: Status 404 returned error can't find the container with id 31dd8b0c4bfd1b74a298b4ae4750ffeb653b50bc59da82ab548a9244e3c34b86 Oct 30 17:36:09.777162 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.777131 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:36:09.777238 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.777199 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:36:09.777238 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.777231 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:36:09.803296 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:36:09.803267 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84fbd1da_e017_490a_8db0_1c9555ceb3b4.slice/crio-b7581708f371f2316637328221ae82ce23e5d379a6a1c4edd47ffd1b3862e4ac WatchSource:0}: Error finding container b7581708f371f2316637328221ae82ce23e5d379a6a1c4edd47ffd1b3862e4ac: Status 404 returned error can't find the container with id b7581708f371f2316637328221ae82ce23e5d379a6a1c4edd47ffd1b3862e4ac Oct 30 17:36:09.803994 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.803954 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016f1d-5d984ddf9-sjb6b"] Oct 30 17:36:09.805091 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:36:09.805067 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17d99e83_45e9_446a_9525_ee96fe9ff19a.slice/crio-3df6f725988dd7af3eb2890341090f1c3aa2b3aa18cf7befb5299c2159b9e1cf WatchSource:0}: Error finding container 3df6f725988dd7af3eb2890341090f1c3aa2b3aa18cf7befb5299c2159b9e1cf: Status 404 returned error can't find the container with id 3df6f725988dd7af3eb2890341090f1c3aa2b3aa18cf7befb5299c2159b9e1cf Oct 30 17:36:09.808517 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:09.808497 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-016f1d-78b77d5dc4-qxvr6"] Oct 30 17:36:10.161078 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:10.161041 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016f1d-5d984ddf9-sjb6b" event={"ID":"84fbd1da-e017-490a-8db0-1c9555ceb3b4","Type":"ContainerStarted","Data":"36b2412a074d7f4bb195cd58ed4464844915709c73526c7bc87d83b910c6b3ae"} Oct 30 17:36:10.161376 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:10.161129 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016f1d-5d984ddf9-sjb6b" event={"ID":"84fbd1da-e017-490a-8db0-1c9555ceb3b4","Type":"ContainerStarted","Data":"b7581708f371f2316637328221ae82ce23e5d379a6a1c4edd47ffd1b3862e4ac"} Oct 30 17:36:10.161376 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:10.161158 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016f1d-5d984ddf9-sjb6b" Oct 30 17:36:10.163283 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:10.163254 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016f1d-78b77d5dc4-qxvr6" event={"ID":"17d99e83-45e9-446a-9525-ee96fe9ff19a","Type":"ContainerStarted","Data":"46445a17a3acec951ad67c73c3b719c4b0ef223362685f63f01eb1717020fcd1"} Oct 30 17:36:10.163417 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:10.163306 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016f1d-78b77d5dc4-qxvr6" event={"ID":"17d99e83-45e9-446a-9525-ee96fe9ff19a","Type":"ContainerStarted","Data":"3df6f725988dd7af3eb2890341090f1c3aa2b3aa18cf7befb5299c2159b9e1cf"} Oct 30 17:36:10.163417 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:10.163336 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016f1d-78b77d5dc4-qxvr6" Oct 30 17:36:10.164986 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:10.164955 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" event={"ID":"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e","Type":"ContainerStarted","Data":"56e8be9eeb4e8291cd6e6ad01953970ad164de022a156f9d86511a974b13cfae"} Oct 30 17:36:10.165131 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:10.165011 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" event={"ID":"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e","Type":"ContainerStarted","Data":"31dd8b0c4bfd1b74a298b4ae4750ffeb653b50bc59da82ab548a9244e3c34b86"} Oct 30 17:36:10.204843 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:10.204786 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016f1d-5d984ddf9-sjb6b" podStartSLOduration=2.204770232 podStartE2EDuration="2.204770232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:10.202982082 +0000 UTC m=+1428.255082008" watchObservedRunningTime="2026-10-30 17:36:10.204770232 +0000 UTC m=+1428.256870163" Oct 30 17:36:10.236747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:10.236714 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:10.242591 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:10.242568 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:10.266178 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:10.266123 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" podStartSLOduration=2.26610824 podStartE2EDuration="2.26610824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:10.262738655 +0000 UTC m=+1428.314838586" watchObservedRunningTime="2026-10-30 17:36:10.26610824 +0000 UTC m=+1428.318208244" Oct 30 17:36:10.318221 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:10.318160 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016f1d-78b77d5dc4-qxvr6" podStartSLOduration=2.3181347580000002 podStartE2EDuration="2.318134758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:10.313112501 +0000 UTC m=+1428.365212433" watchObservedRunningTime="2026-10-30 17:36:10.318134758 +0000 UTC m=+1428.370234689" Oct 30 17:36:11.185644 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:11.185474 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:11.187966 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:11.187940 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:36:11.798010 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:11.797976 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7bb2e69d-54765d4d8c-kvxq7" Oct 30 17:36:14.936960 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:14.936925 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8d1062-595d67866c-92d2c"] Oct 30 17:36:14.942992 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:14.942965 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8d1062-595d67866c-92d2c" Oct 30 17:36:14.960946 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:14.960916 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-8d1062-dockercfg-pb2fl\"" Oct 30 17:36:14.963240 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:14.963215 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8d1062-595d67866c-92d2c"] Oct 30 17:36:14.996807 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:14.996777 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzvgj\" (UniqueName: \"kubernetes.io/projected/1a68dfa4-44fa-40d0-bf51-ecac52a252d7-kube-api-access-jzvgj\") pod \"payload-pre-processing-e2e-shared-a-w6-8d1062-595d67866c-92d2c\" (UID: \"1a68dfa4-44fa-40d0-bf51-ecac52a252d7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8d1062-595d67866c-92d2c" Oct 30 17:36:14.997060 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:14.997031 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a68dfa4-44fa-40d0-bf51-ecac52a252d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-8d1062-595d67866c-92d2c\" (UID: \"1a68dfa4-44fa-40d0-bf51-ecac52a252d7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8d1062-595d67866c-92d2c" Oct 30 17:36:15.057893 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:15.048145 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-8d1062\"" Oct 30 17:36:15.148888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:15.108021 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzvgj\" (UniqueName: \"kubernetes.io/projected/1a68dfa4-44fa-40d0-bf51-ecac52a252d7-kube-api-access-jzvgj\") pod \"payload-pre-processing-e2e-shared-a-w6-8d1062-595d67866c-92d2c\" (UID: \"1a68dfa4-44fa-40d0-bf51-ecac52a252d7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8d1062-595d67866c-92d2c" Oct 30 17:36:15.170358 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:15.170316 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a68dfa4-44fa-40d0-bf51-ecac52a252d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-8d1062-595d67866c-92d2c\" (UID: \"1a68dfa4-44fa-40d0-bf51-ecac52a252d7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8d1062-595d67866c-92d2c" Oct 30 17:36:15.186893 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:15.171828 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a68dfa4-44fa-40d0-bf51-ecac52a252d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-8d1062-595d67866c-92d2c\" (UID: \"1a68dfa4-44fa-40d0-bf51-ecac52a252d7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8d1062-595d67866c-92d2c" Oct 30 17:36:15.187215 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:15.173564 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzvgj\" (UniqueName: \"kubernetes.io/projected/1a68dfa4-44fa-40d0-bf51-ecac52a252d7-kube-api-access-jzvgj\") pod \"payload-pre-processing-e2e-shared-a-w6-8d1062-595d67866c-92d2c\" (UID: \"1a68dfa4-44fa-40d0-bf51-ecac52a252d7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8d1062-595d67866c-92d2c" Oct 30 17:36:15.266204 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:15.266166 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8d1062-595d67866c-92d2c" Oct 30 17:36:15.768355 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:15.768325 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8d1062-595d67866c-92d2c"] Oct 30 17:36:15.768558 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:36:15.768535 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a68dfa4_44fa_40d0_bf51_ecac52a252d7.slice/crio-ef1e98a322254af16ad8449380fb8707cadf686b3214837f5aa592f110adc0ed WatchSource:0}: Error finding container ef1e98a322254af16ad8449380fb8707cadf686b3214837f5aa592f110adc0ed: Status 404 returned error can't find the container with id ef1e98a322254af16ad8449380fb8707cadf686b3214837f5aa592f110adc0ed Oct 30 17:36:16.259590 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:16.259548 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8d1062-595d67866c-92d2c" event={"ID":"1a68dfa4-44fa-40d0-bf51-ecac52a252d7","Type":"ContainerStarted","Data":"6ae936b4b049bef952eb1349149bcbfa65ffe5c99b8093ef79a93d70f8be7d62"} Oct 30 17:36:16.259998 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:16.259611 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8d1062-595d67866c-92d2c" event={"ID":"1a68dfa4-44fa-40d0-bf51-ecac52a252d7","Type":"ContainerStarted","Data":"ef1e98a322254af16ad8449380fb8707cadf686b3214837f5aa592f110adc0ed"} Oct 30 17:36:16.259998 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:16.259658 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8d1062-595d67866c-92d2c" Oct 30 17:36:16.306384 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:16.306337 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8d1062-595d67866c-92d2c" podStartSLOduration=2.306323237 podStartE2EDuration="2.306323237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:16.30296847 +0000 UTC m=+1434.355068459" watchObservedRunningTime="2026-10-30 17:36:16.306323237 +0000 UTC m=+1434.358423167" Oct 30 17:36:18.482888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:18.453835 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:36:21.226309 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:21.226276 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016f1d-78b77d5dc4-qxvr6" Oct 30 17:36:21.228570 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:21.228545 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016f1d-5d984ddf9-sjb6b" Oct 30 17:36:21.667799 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:21.662370 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:22.061944 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:22.061430 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 17:36:22.167154 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:22.125182 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:22.321232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:22.321126 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:22.361124 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:22.361077 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 17:36:22.361709 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:22.361675 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" podUID="d174a74f-7205-4020-94b0-e30652c9b721" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:22.388101 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:22.388055 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 17:36:22.417150 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:22.417099 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:22.419366 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:22.419328 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:22.446877 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:22.446805 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:22.685885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:22.685576 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:22.686098 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:22.686068 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:36:23.141946 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:23.137004 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-zv8m5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:36:23.141946 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:23.137090 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" podUID="0a692092-9a09-4a93-b285-675f8268eb68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:23.141946 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:23.137623 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:23.141946 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:23.138082 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:36:23.493323 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:23.493220 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:23.589246 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:23.586338 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" podUID="d174a74f-7205-4020-94b0-e30652c9b721" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 30 17:36:23.917684 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:23.917643 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:23.934527 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:23.926498 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" Oct 30 17:36:23.945493 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:23.945458 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:36:23.951050 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:23.950918 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 17:36:23.951584 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:23.951562 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:36:23.952511 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:23.951939 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 17:36:23.954250 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:23.953388 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:23.955403 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:23.955387 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:36:23.955574 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:23.955560 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:36:23.956433 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:23.953734 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 17:36:23.972886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:23.971358 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:36:23.998598 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:23.748323 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:24.221308 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:24.221093 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 17:36:24.652893 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:24.646660 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" podUID="d174a74f-7205-4020-94b0-e30652c9b721" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:24.940935 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:24.938970 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:25.003886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:25.003026 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ms6c5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:36:25.003886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:25.003085 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" podUID="f835cdef-6be0-4f7d-b4c1-6e5a771e8616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:25.128382 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:25.128338 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:25.290945 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:25.228432 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:36:25.567779 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:25.567702 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:36:25.568031 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:25.199015 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:25.615060 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:25.313522 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:25.615060 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:25.198980 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 17:36:25.627616 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:25.313562 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded" Oct 30 17:36:25.761719 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:25.761674 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 17:36:26.050156 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:26.050097 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 17:36:26.093978 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:25.649768 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" podUID="d174a74f-7205-4020-94b0-e30652c9b721" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 30 17:36:26.133943 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:25.417241 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 17:36:26.284027 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:26.283985 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:26.345939 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:26.324151 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-st6hg" podUID="393bb669-2192-4775-b63c-f96a2aba0cd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:26.484886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:26.363602 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 17:36:26.632940 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:26.576482 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 17:36:26.652289 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:26.643925 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 17:36:27.070937 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:27.066530 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:27.105091 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:27.079528 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" podUID="d174a74f-7205-4020-94b0-e30652c9b721" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:27.134891 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:27.132851 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:27.243888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:27.237194 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-df8b20-69845c8495-f4mq5"] Oct 30 17:36:27.267898 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:27.264947 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:27.290805 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:27.290759 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:27.532963 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:27.529020 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:27.598410 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:27.598379 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-df8b20-69845c8495-f4mq5"] Oct 30 17:36:27.603213 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:27.598846 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:36:27.603761 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:27.603737 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:36:27.604003 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:27.603986 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8d1062-595d67866c-92d2c" Oct 30 17:36:27.621638 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:27.621611 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df8b20-69845c8495-f4mq5" Oct 30 17:36:27.635315 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:27.635291 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:36:27.655051 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:27.652400 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-df8b20\"" Oct 30 17:36:27.673236 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:27.670221 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:27.682234 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:27.682184 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:27.688965 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:27.688903 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:27.731114 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:27.718756 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-df8b20-dockercfg-tsmrd\"" Oct 30 17:36:27.763748 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:27.761674 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 17:36:27.982961 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:27.982921 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjb6n\" (UniqueName: \"kubernetes.io/projected/631b7466-f017-4430-89ee-f62b93d000bb-kube-api-access-xjb6n\") pod \"payload-pre-processing-e2e-api-a-df8b20-69845c8495-f4mq5\" (UID: \"631b7466-f017-4430-89ee-f62b93d000bb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-df8b20-69845c8495-f4mq5" Oct 30 17:36:27.983845 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:27.983824 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/631b7466-f017-4430-89ee-f62b93d000bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-df8b20-69845c8495-f4mq5\" (UID: \"631b7466-f017-4430-89ee-f62b93d000bb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-df8b20-69845c8495-f4mq5" Oct 30 17:36:28.143933 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:28.133991 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" podUID="d174a74f-7205-4020-94b0-e30652c9b721" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 30 17:36:28.143933 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:28.143269 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 17:36:28.156933 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:28.148967 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 17:36:28.246951 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:28.230518 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjb6n\" (UniqueName: \"kubernetes.io/projected/631b7466-f017-4430-89ee-f62b93d000bb-kube-api-access-xjb6n\") pod \"payload-pre-processing-e2e-api-a-df8b20-69845c8495-f4mq5\" (UID: \"631b7466-f017-4430-89ee-f62b93d000bb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-df8b20-69845c8495-f4mq5" Oct 30 17:36:28.247598 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:28.247561 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/631b7466-f017-4430-89ee-f62b93d000bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-df8b20-69845c8495-f4mq5\" (UID: \"631b7466-f017-4430-89ee-f62b93d000bb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-df8b20-69845c8495-f4mq5" Oct 30 17:36:28.327943 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:28.323128 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/631b7466-f017-4430-89ee-f62b93d000bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-df8b20-69845c8495-f4mq5\" (UID: \"631b7466-f017-4430-89ee-f62b93d000bb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-df8b20-69845c8495-f4mq5" Oct 30 17:36:28.609943 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:28.583430 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 17:36:28.719884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:28.719285 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:28.746792 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:28.746755 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjb6n\" (UniqueName: \"kubernetes.io/projected/631b7466-f017-4430-89ee-f62b93d000bb-kube-api-access-xjb6n\") pod \"payload-pre-processing-e2e-api-a-df8b20-69845c8495-f4mq5\" (UID: \"631b7466-f017-4430-89ee-f62b93d000bb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-df8b20-69845c8495-f4mq5" Oct 30 17:36:28.771855 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:28.758968 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:28.772302 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:28.759135 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 17:36:28.933123 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:28.933039 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df8b20-69845c8495-f4mq5" Oct 30 17:36:29.165119 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:29.165074 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:29.166049 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:29.165655 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:29.177085 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:29.176770 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:36:29.200081 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:29.199087 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" podUID="d174a74f-7205-4020-94b0-e30652c9b721" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:29.257895 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:29.254749 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" Oct 30 17:36:29.455517 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:29.455428 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 17:36:29.455921 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:29.455800 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 17:36:29.456388 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:29.456368 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 17:36:29.484889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:29.458374 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 17:36:30.518996 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:30.518966 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-df8b20-69845c8495-f4mq5"] Oct 30 17:36:30.944057 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:30.944019 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df8b20-69845c8495-f4mq5" event={"ID":"631b7466-f017-4430-89ee-f62b93d000bb","Type":"ContainerStarted","Data":"3d7b15492caa77bf4f405797ca7acd32570d24e393ee3004f826f97e5080c10b"} Oct 30 17:36:30.944266 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:30.944080 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df8b20-69845c8495-f4mq5" event={"ID":"631b7466-f017-4430-89ee-f62b93d000bb","Type":"ContainerStarted","Data":"ffe76e440482f074c1931c78320016d3b6ab1ee713c760687313979027476d0c"} Oct 30 17:36:30.944266 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:30.944109 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df8b20-69845c8495-f4mq5" Oct 30 17:36:31.016963 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:31.016905 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df8b20-69845c8495-f4mq5" podStartSLOduration=5.016886681 podStartE2EDuration="5.016886681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:31.013112246 +0000 UTC m=+1449.065212178" watchObservedRunningTime="2026-10-30 17:36:31.016886681 +0000 UTC m=+1449.068986612" Oct 30 17:36:35.400236 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:35.400199 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd"] Oct 30 17:36:35.433718 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:35.433690 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd" Oct 30 17:36:35.471681 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:35.471645 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd"] Oct 30 17:36:35.472499 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:35.472477 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-016f1d-serving-cert\"" Oct 30 17:36:35.488253 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:35.488214 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57095c88-ac35-4dc9-b3da-4237fcf6a752-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd\" (UID: \"57095c88-ac35-4dc9-b3da-4237fcf6a752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd" Oct 30 17:36:35.488435 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:35.488311 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57095c88-ac35-4dc9-b3da-4237fcf6a752-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd\" (UID: \"57095c88-ac35-4dc9-b3da-4237fcf6a752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd" Oct 30 17:36:35.488435 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:35.488409 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7xrn\" (UniqueName: \"kubernetes.io/projected/57095c88-ac35-4dc9-b3da-4237fcf6a752-kube-api-access-n7xrn\") pod \"maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd\" (UID: \"57095c88-ac35-4dc9-b3da-4237fcf6a752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd" Oct 30 17:36:35.488567 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:35.488534 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57095c88-ac35-4dc9-b3da-4237fcf6a752-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd\" (UID: \"57095c88-ac35-4dc9-b3da-4237fcf6a752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd" Oct 30 17:36:35.635887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:35.635654 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57095c88-ac35-4dc9-b3da-4237fcf6a752-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd\" (UID: \"57095c88-ac35-4dc9-b3da-4237fcf6a752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd" Oct 30 17:36:35.635887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:35.635711 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7xrn\" (UniqueName: \"kubernetes.io/projected/57095c88-ac35-4dc9-b3da-4237fcf6a752-kube-api-access-n7xrn\") pod \"maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd\" (UID: \"57095c88-ac35-4dc9-b3da-4237fcf6a752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd" Oct 30 17:36:35.635887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:35.635889 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57095c88-ac35-4dc9-b3da-4237fcf6a752-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd\" (UID: \"57095c88-ac35-4dc9-b3da-4237fcf6a752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd" Oct 30 17:36:35.636185 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:35.636138 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57095c88-ac35-4dc9-b3da-4237fcf6a752-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd\" (UID: \"57095c88-ac35-4dc9-b3da-4237fcf6a752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd" Oct 30 17:36:35.651670 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:35.651591 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57095c88-ac35-4dc9-b3da-4237fcf6a752-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd\" (UID: \"57095c88-ac35-4dc9-b3da-4237fcf6a752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd" Oct 30 17:36:35.686062 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:35.686019 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57095c88-ac35-4dc9-b3da-4237fcf6a752-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd\" (UID: \"57095c88-ac35-4dc9-b3da-4237fcf6a752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd" Oct 30 17:36:35.707201 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:35.705921 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7xrn\" (UniqueName: \"kubernetes.io/projected/57095c88-ac35-4dc9-b3da-4237fcf6a752-kube-api-access-n7xrn\") pod \"maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd\" (UID: \"57095c88-ac35-4dc9-b3da-4237fcf6a752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd" Oct 30 17:36:35.723832 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:35.723797 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57095c88-ac35-4dc9-b3da-4237fcf6a752-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd\" (UID: \"57095c88-ac35-4dc9-b3da-4237fcf6a752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd" Oct 30 17:36:35.770841 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:35.767438 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd" Oct 30 17:36:36.016379 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:36:36.016332 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:36:36.131384 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:36:36.131353 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57095c88_ac35_4dc9_b3da_4237fcf6a752.slice/crio-766e1225921b1c60dd45f34c38e8a12f24c481f67a3dc1e269900d20f503a686 WatchSource:0}: Error finding container 766e1225921b1c60dd45f34c38e8a12f24c481f67a3dc1e269900d20f503a686: Status 404 returned error can't find the container with id 766e1225921b1c60dd45f34c38e8a12f24c481f67a3dc1e269900d20f503a686 Oct 30 17:36:36.132910 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:36.132886 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd"] Oct 30 17:36:37.062928 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:37.061290 4965 generic.go:356] "Generic (PLEG): container finished" podID="6497147f-bad2-434c-a6e4-45b247b75fc2" containerID="1ec22adeee1cac2f41e7428a64b322f23700d26c42777a02438683ab45754b0f" exitCode=0 Oct 30 17:36:37.062928 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:37.061361 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" event={"ID":"6497147f-bad2-434c-a6e4-45b247b75fc2","Type":"ContainerDied","Data":"1ec22adeee1cac2f41e7428a64b322f23700d26c42777a02438683ab45754b0f"} Oct 30 17:36:37.062928 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:37.061896 4965 scope.go:122] "RemoveContainer" containerID="1ec22adeee1cac2f41e7428a64b322f23700d26c42777a02438683ab45754b0f" Oct 30 17:36:37.084888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:37.076439 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:36:37.084888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:37.076551 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:36:37.084888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:37.076601 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:36:37.084888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:37.080481 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd" event={"ID":"57095c88-ac35-4dc9-b3da-4237fcf6a752","Type":"ContainerStarted","Data":"aa51712a2af7641a31cfc6b0bd47b1a8a5c877852a79abf0ebf86b5332a4f655"} Oct 30 17:36:37.084888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:37.080558 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd" event={"ID":"57095c88-ac35-4dc9-b3da-4237fcf6a752","Type":"ContainerStarted","Data":"766e1225921b1c60dd45f34c38e8a12f24c481f67a3dc1e269900d20f503a686"} Oct 30 17:36:37.084888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:37.081403 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd" Oct 30 17:36:37.154084 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:37.142764 4965 generic.go:356] "Generic (PLEG): container finished" podID="8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d" containerID="778ff37a1424c5358b1ceaa72ddecac6f3bb1906ce75db337be7471a380bc0d0" exitCode=0 Oct 30 17:36:37.154084 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:37.142849 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" event={"ID":"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d","Type":"ContainerDied","Data":"778ff37a1424c5358b1ceaa72ddecac6f3bb1906ce75db337be7471a380bc0d0"} Oct 30 17:36:37.154084 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:37.151166 4965 scope.go:122] "RemoveContainer" containerID="778ff37a1424c5358b1ceaa72ddecac6f3bb1906ce75db337be7471a380bc0d0" Oct 30 17:36:37.160343 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:37.158633 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:36:37.160343 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:37.158920 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:36:37.160343 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:37.159326 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:36:37.240605 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:37.240298 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd" podStartSLOduration=1.8107058 podStartE2EDuration="2.240278809s" totalImagesPullingTime="429.573009ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:37.226302578 +0000 UTC m=+1455.278402509" watchObservedRunningTime="2026-10-30 17:36:37.240278809 +0000 UTC m=+1455.292378741" Oct 30 17:36:38.152073 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:38.152036 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" event={"ID":"6497147f-bad2-434c-a6e4-45b247b75fc2","Type":"ContainerStarted","Data":"5e32b8689c8d76a9b29646ab656bc54e8c6b8bbbfa5f1d748ed5f0bbca51e53e"} Oct 30 17:36:38.154053 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:38.154030 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" event={"ID":"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d","Type":"ContainerStarted","Data":"f4e4b8cd86263b1c642bdd347d9e0ffd208835c744a40828bf221b1da7ab1546"} Oct 30 17:36:38.352384 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:38.352351 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:36:38.353664 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:38.353635 4965 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 17:36:38.353794 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:38.353688 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" podUID="8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 17:36:38.838984 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:38.838951 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:36:38.839697 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:38.839671 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 17:36:38.839797 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:38.839739 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" podUID="6497147f-bad2-434c-a6e4-45b247b75fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 17:36:39.352461 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:39.352432 4965 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 17:36:39.352874 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:39.352481 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" podUID="8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 17:36:39.839101 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:39.839073 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 17:36:39.839336 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:39.839315 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" podUID="6497147f-bad2-434c-a6e4-45b247b75fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 17:36:40.354243 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:40.354207 4965 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 17:36:40.354752 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:40.354263 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" podUID="8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 17:36:40.839768 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:40.839730 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 17:36:40.839959 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:40.839784 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" podUID="6497147f-bad2-434c-a6e4-45b247b75fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 17:36:41.219241 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:41.218001 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:36:41.351750 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:41.351715 4965 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 17:36:41.351933 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:41.351765 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" podUID="8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 17:36:41.839554 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:41.839521 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 17:36:41.840089 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:41.839616 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" podUID="6497147f-bad2-434c-a6e4-45b247b75fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 17:36:41.955839 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:41.955810 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df8b20-69845c8495-f4mq5" Oct 30 17:36:42.037377 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.037341 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2"] Oct 30 17:36:42.267009 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.266913 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:42.579642 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.518996 4965 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 17:36:42.579642 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.519091 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" podUID="8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 17:36:42.579642 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.565936 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-d53ec1-openshift-default-dockercfg-2q58v\"" Oct 30 17:36:42.677888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.664557 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-workload-certs\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:42.677888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.664604 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551401b3-f7a0-4b52-afac-f68219c56970-istio-podinfo\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:42.677888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.664683 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551401b3-f7a0-4b52-afac-f68219c56970-istiod-ca-cert\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:42.677888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.664791 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-credential-socket\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:42.677888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.664839 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551401b3-f7a0-4b52-afac-f68219c56970-istio-token\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:42.677888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.664893 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-workload-socket\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:42.677888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.665006 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phzfg\" (UniqueName: \"kubernetes.io/projected/551401b3-f7a0-4b52-afac-f68219c56970-kube-api-access-phzfg\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:42.677888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.665093 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-istio-envoy\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:42.677888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.665176 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-istio-data\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:42.789033 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.783925 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-workload-certs\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:42.793146 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.788939 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-workload-certs\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:42.797353 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.794823 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551401b3-f7a0-4b52-afac-f68219c56970-istio-podinfo\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:42.797909 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.797886 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551401b3-f7a0-4b52-afac-f68219c56970-istiod-ca-cert\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:42.800040 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.800011 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-credential-socket\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:42.800148 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.800073 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551401b3-f7a0-4b52-afac-f68219c56970-istio-token\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:42.800148 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.800116 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-workload-socket\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:42.800288 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.800216 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phzfg\" (UniqueName: \"kubernetes.io/projected/551401b3-f7a0-4b52-afac-f68219c56970-kube-api-access-phzfg\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:42.800351 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.800292 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-istio-envoy\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:42.800408 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.800370 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-istio-data\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:42.803882 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.800943 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-istio-data\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:42.803882 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.801193 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-credential-socket\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:42.803882 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.801750 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-workload-socket\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:42.810764 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.810732 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551401b3-f7a0-4b52-afac-f68219c56970-istiod-ca-cert\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:42.880846 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.880357 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 17:36:42.880846 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.880417 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" podUID="6497147f-bad2-434c-a6e4-45b247b75fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 17:36:42.883722 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.883694 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2"] Oct 30 17:36:42.929886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.927425 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-istio-envoy\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:42.929886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.927761 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551401b3-f7a0-4b52-afac-f68219c56970-istio-token\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:42.929886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.927997 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551401b3-f7a0-4b52-afac-f68219c56970-istio-podinfo\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:42.932434 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:42.932407 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phzfg\" (UniqueName: \"kubernetes.io/projected/551401b3-f7a0-4b52-afac-f68219c56970-kube-api-access-phzfg\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:43.043781 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.043751 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k"] Oct 30 17:36:43.048732 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.048705 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" Oct 30 17:36:43.122327 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.122293 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-81e9336e\"/\"default-dockercfg-nzlcq\"" Oct 30 17:36:43.123972 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.122626 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-81e9336e\"/\"kube-root-ca.crt\"" Oct 30 17:36:43.124156 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.122750 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-81e9336e\"/\"e2e-lifecycle-model-81e9336e-kserve-self-signed-certs\"" Oct 30 17:36:43.124156 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.122800 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-81e9336e\"/\"openshift-service-ca.crt\"" Oct 30 17:36:43.124345 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.123969 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7715de9f-65c1-44fb-b83a-692ec07b30a8-model-cache\") pod \"e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k\" (UID: \"7715de9f-65c1-44fb-b83a-692ec07b30a8\") " pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" Oct 30 17:36:43.124345 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.124231 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7715de9f-65c1-44fb-b83a-692ec07b30a8-tls-certs\") pod \"e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k\" (UID: \"7715de9f-65c1-44fb-b83a-692ec07b30a8\") " pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" Oct 30 17:36:43.124345 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.124317 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7715de9f-65c1-44fb-b83a-692ec07b30a8-tmp-dir\") pod \"e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k\" (UID: \"7715de9f-65c1-44fb-b83a-692ec07b30a8\") " pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" Oct 30 17:36:43.124499 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.124406 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59qfj\" (UniqueName: \"kubernetes.io/projected/7715de9f-65c1-44fb-b83a-692ec07b30a8-kube-api-access-59qfj\") pod \"e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k\" (UID: \"7715de9f-65c1-44fb-b83a-692ec07b30a8\") " pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" Oct 30 17:36:43.124499 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.124446 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7715de9f-65c1-44fb-b83a-692ec07b30a8-dshm\") pod \"e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k\" (UID: \"7715de9f-65c1-44fb-b83a-692ec07b30a8\") " pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" Oct 30 17:36:43.124499 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.124483 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7715de9f-65c1-44fb-b83a-692ec07b30a8-home\") pod \"e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k\" (UID: \"7715de9f-65c1-44fb-b83a-692ec07b30a8\") " pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" Oct 30 17:36:43.128520 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.128445 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k"] Oct 30 17:36:43.139339 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.139287 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:43.211160 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.211125 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd" Oct 30 17:36:43.231197 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.229336 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7715de9f-65c1-44fb-b83a-692ec07b30a8-tmp-dir\") pod \"e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k\" (UID: \"7715de9f-65c1-44fb-b83a-692ec07b30a8\") " pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" Oct 30 17:36:43.231197 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.229435 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59qfj\" (UniqueName: \"kubernetes.io/projected/7715de9f-65c1-44fb-b83a-692ec07b30a8-kube-api-access-59qfj\") pod \"e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k\" (UID: \"7715de9f-65c1-44fb-b83a-692ec07b30a8\") " pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" Oct 30 17:36:43.231197 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.229473 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7715de9f-65c1-44fb-b83a-692ec07b30a8-dshm\") pod \"e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k\" (UID: \"7715de9f-65c1-44fb-b83a-692ec07b30a8\") " pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" Oct 30 17:36:43.231197 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.229507 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7715de9f-65c1-44fb-b83a-692ec07b30a8-home\") pod \"e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k\" (UID: \"7715de9f-65c1-44fb-b83a-692ec07b30a8\") " pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" Oct 30 17:36:43.231197 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.229766 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7715de9f-65c1-44fb-b83a-692ec07b30a8-model-cache\") pod \"e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k\" (UID: \"7715de9f-65c1-44fb-b83a-692ec07b30a8\") " pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" Oct 30 17:36:43.231197 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.229809 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7715de9f-65c1-44fb-b83a-692ec07b30a8-tls-certs\") pod \"e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k\" (UID: \"7715de9f-65c1-44fb-b83a-692ec07b30a8\") " pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" Oct 30 17:36:43.231197 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.230851 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7715de9f-65c1-44fb-b83a-692ec07b30a8-tmp-dir\") pod \"e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k\" (UID: \"7715de9f-65c1-44fb-b83a-692ec07b30a8\") " pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" Oct 30 17:36:43.232381 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.231325 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7715de9f-65c1-44fb-b83a-692ec07b30a8-home\") pod \"e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k\" (UID: \"7715de9f-65c1-44fb-b83a-692ec07b30a8\") " pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" Oct 30 17:36:43.232381 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.231560 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7715de9f-65c1-44fb-b83a-692ec07b30a8-model-cache\") pod \"e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k\" (UID: \"7715de9f-65c1-44fb-b83a-692ec07b30a8\") " pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" Oct 30 17:36:43.232381 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.231618 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:36:43.238190 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.238163 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7715de9f-65c1-44fb-b83a-692ec07b30a8-dshm\") pod \"e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k\" (UID: \"7715de9f-65c1-44fb-b83a-692ec07b30a8\") " pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" Oct 30 17:36:43.242764 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.242741 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7715de9f-65c1-44fb-b83a-692ec07b30a8-tls-certs\") pod \"e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k\" (UID: \"7715de9f-65c1-44fb-b83a-692ec07b30a8\") " pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" Oct 30 17:36:43.254852 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.254826 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59qfj\" (UniqueName: \"kubernetes.io/projected/7715de9f-65c1-44fb-b83a-692ec07b30a8-kube-api-access-59qfj\") pod \"e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k\" (UID: \"7715de9f-65c1-44fb-b83a-692ec07b30a8\") " pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" Oct 30 17:36:43.353163 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.353125 4965 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 17:36:43.353292 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.353200 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" podUID="8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 17:36:43.440408 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.439389 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" Oct 30 17:36:43.844350 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.843183 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 17:36:43.844350 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.843241 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" podUID="6497147f-bad2-434c-a6e4-45b247b75fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 17:36:43.883177 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.857305 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2"] Oct 30 17:36:43.933624 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.933056 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:36:43.933624 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.933151 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:36:43.933624 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:43.933190 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:36:44.353795 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:44.353764 4965 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 17:36:44.354176 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:44.354136 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" podUID="8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 17:36:44.502897 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:36:44.502852 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7715de9f_65c1_44fb_b83a_692ec07b30a8.slice/crio-30d6246d4d38905ab61df06b82fe39ef147d336df57fbfbe648a27571ca5d1f2 WatchSource:0}: Error finding container 30d6246d4d38905ab61df06b82fe39ef147d336df57fbfbe648a27571ca5d1f2: Status 404 returned error can't find the container with id 30d6246d4d38905ab61df06b82fe39ef147d336df57fbfbe648a27571ca5d1f2 Oct 30 17:36:44.504962 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:44.504847 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k"] Oct 30 17:36:44.844185 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:44.844156 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:36:44.845661 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:44.845638 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 17:36:44.913211 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:44.912030 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" event={"ID":"551401b3-f7a0-4b52-afac-f68219c56970","Type":"ContainerStarted","Data":"ff1c89d80bdcd2394bd9266df2d3fa4dc1d7f322448d97b16ae1a5b52e64c535"} Oct 30 17:36:44.913211 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:44.912099 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" event={"ID":"551401b3-f7a0-4b52-afac-f68219c56970","Type":"ContainerStarted","Data":"3158f58d69dc036305bd6302c4a7f52ae7a652617911902306efa978ceddc394"} Oct 30 17:36:44.919847 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:44.919815 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" event={"ID":"7715de9f-65c1-44fb-b83a-692ec07b30a8","Type":"ContainerStarted","Data":"c3e8dfc4684e450367e82d52bed9b9ceb0c6722825dce3da857d0ccd1a2ec854"} Oct 30 17:36:44.920235 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:44.920207 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" event={"ID":"7715de9f-65c1-44fb-b83a-692ec07b30a8","Type":"ContainerStarted","Data":"30d6246d4d38905ab61df06b82fe39ef147d336df57fbfbe648a27571ca5d1f2"} Oct 30 17:36:45.018059 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:45.017976 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" podStartSLOduration=4.017957167 podStartE2EDuration="4.017957167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:45.016083857 +0000 UTC m=+1463.068183791" watchObservedRunningTime="2026-10-30 17:36:45.017957167 +0000 UTC m=+1463.070057097" Oct 30 17:36:45.063421 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:45.063359 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" podStartSLOduration=2.063340223 podStartE2EDuration="2.063340223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:45.060717317 +0000 UTC m=+1463.112817247" watchObservedRunningTime="2026-10-30 17:36:45.063340223 +0000 UTC m=+1463.115440159" Oct 30 17:36:45.140085 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:45.139997 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:45.145478 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:45.145456 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:45.374885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:45.371162 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:36:45.374885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:45.374416 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:36:45.460848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:45.460173 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" Oct 30 17:36:45.935705 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:45.935672 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:45.938401 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:45.938376 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:36:50.653892 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:50.648544 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:36:53.460952 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:53.460921 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" Oct 30 17:36:53.482267 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:53.482232 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" Oct 30 17:36:54.711887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:54.708974 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ms6c5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:36:54.711887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:54.709043 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" podUID="f835cdef-6be0-4f7d-b4c1-6e5a771e8616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:54.947888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:54.934006 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:55.028952 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:55.027981 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:55.060888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:55.057016 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:55.151924 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:55.139092 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded" Oct 30 17:36:55.181890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:55.180749 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:55.255592 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:55.254996 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:55.290050 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:55.289947 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:55.327895 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:55.322970 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" podUID="7715de9f-65c1-44fb-b83a-692ec07b30a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:55.909723 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:36:55.909694 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" Oct 30 17:37:05.999831 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:37:05.999013 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:37:22.327613 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:22.327581 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 17:37:22.328253 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:22.327816 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 17:37:31.148258 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:31.148218 4965 generic.go:356] "Generic (PLEG): container finished" podID="82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e" containerID="56e8be9eeb4e8291cd6e6ad01953970ad164de022a156f9d86511a974b13cfae" exitCode=0 Oct 30 17:37:31.148754 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:31.148344 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" event={"ID":"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e","Type":"ContainerDied","Data":"56e8be9eeb4e8291cd6e6ad01953970ad164de022a156f9d86511a974b13cfae"} Oct 30 17:37:31.148810 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:31.148762 4965 scope.go:122] "RemoveContainer" containerID="56e8be9eeb4e8291cd6e6ad01953970ad164de022a156f9d86511a974b13cfae" Oct 30 17:37:31.150536 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:31.150504 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:37:31.150620 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:31.150576 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:37:31.150620 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:31.150608 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:37:32.254942 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:32.254425 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" event={"ID":"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e","Type":"ContainerStarted","Data":"f298491a0f8a7f978b59eca8e56c6796ee878ba49feb6f86491d7a6834ebe302"} Oct 30 17:37:33.237474 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:33.237439 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:37:33.237640 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:33.237565 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 17:37:33.237640 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:33.237612 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" podUID="82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 17:37:34.237611 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:34.237575 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 17:37:34.238033 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:34.237631 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" podUID="82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 17:37:35.237402 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:35.237370 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 17:37:35.237580 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:35.237435 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" podUID="82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 17:37:36.246651 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:36.246612 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 17:37:36.247439 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:36.247397 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" podUID="82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 17:37:36.328892 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:37:36.313334 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:37:37.237416 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:37.237339 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 17:37:37.237697 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:37.237665 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" podUID="82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 17:37:38.241595 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:38.241565 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:37:38.242063 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:38.242050 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:37:38.242852 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:38.242829 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 17:37:51.234889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:51.233831 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:37:52.433241 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:52.433210 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-4dcdbb-77d699b46c-qm59b"] Oct 30 17:37:52.438136 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:52.438117 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-4dcdbb-77d699b46c-qm59b" Oct 30 17:37:52.464499 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:52.464469 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-4dcdbb-dockercfg-dcsth\"" Oct 30 17:37:52.469499 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:52.469471 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-4dcdbb-77d699b46c-qm59b"] Oct 30 17:37:52.474526 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:52.474502 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-4dcdbb\"" Oct 30 17:37:52.500782 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:52.500755 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dec2e67-e439-47cc-bf7b-b315dae10d2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-4dcdbb-77d699b46c-qm59b\" (UID: \"8dec2e67-e439-47cc-bf7b-b315dae10d2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-4dcdbb-77d699b46c-qm59b" Oct 30 17:37:52.500990 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:52.500807 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srhj6\" (UniqueName: \"kubernetes.io/projected/8dec2e67-e439-47cc-bf7b-b315dae10d2e-kube-api-access-srhj6\") pod \"payload-pre-processing-e2e-shared-b-w6-4dcdbb-77d699b46c-qm59b\" (UID: \"8dec2e67-e439-47cc-bf7b-b315dae10d2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-4dcdbb-77d699b46c-qm59b" Oct 30 17:37:52.615887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:52.610127 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dec2e67-e439-47cc-bf7b-b315dae10d2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-4dcdbb-77d699b46c-qm59b\" (UID: \"8dec2e67-e439-47cc-bf7b-b315dae10d2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-4dcdbb-77d699b46c-qm59b" Oct 30 17:37:52.615887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:52.610247 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srhj6\" (UniqueName: \"kubernetes.io/projected/8dec2e67-e439-47cc-bf7b-b315dae10d2e-kube-api-access-srhj6\") pod \"payload-pre-processing-e2e-shared-b-w6-4dcdbb-77d699b46c-qm59b\" (UID: \"8dec2e67-e439-47cc-bf7b-b315dae10d2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-4dcdbb-77d699b46c-qm59b" Oct 30 17:37:52.664351 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:52.664315 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dec2e67-e439-47cc-bf7b-b315dae10d2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-4dcdbb-77d699b46c-qm59b\" (UID: \"8dec2e67-e439-47cc-bf7b-b315dae10d2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-4dcdbb-77d699b46c-qm59b" Oct 30 17:37:52.710177 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:52.683919 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-4dcdbb-6bcbbcccc6-sbssx"] Oct 30 17:37:52.710177 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:52.690087 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-4dcdbb-6bcbbcccc6-sbssx" Oct 30 17:37:52.710177 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:52.704563 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srhj6\" (UniqueName: \"kubernetes.io/projected/8dec2e67-e439-47cc-bf7b-b315dae10d2e-kube-api-access-srhj6\") pod \"payload-pre-processing-e2e-shared-b-w6-4dcdbb-77d699b46c-qm59b\" (UID: \"8dec2e67-e439-47cc-bf7b-b315dae10d2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-4dcdbb-77d699b46c-qm59b" Oct 30 17:37:52.735289 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:52.712330 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c0c2a4a-8305-4fc2-a440-56d6725b1ba8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-4dcdbb-6bcbbcccc6-sbssx\" (UID: \"5c0c2a4a-8305-4fc2-a440-56d6725b1ba8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-4dcdbb-6bcbbcccc6-sbssx" Oct 30 17:37:52.755393 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:52.755360 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-4dcdbb-77d699b46c-qm59b" Oct 30 17:37:52.904931 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:52.867293 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-4dcdbb-6bcbbcccc6-sbssx"] Oct 30 17:37:52.904931 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:52.883977 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv5d7\" (UniqueName: \"kubernetes.io/projected/5c0c2a4a-8305-4fc2-a440-56d6725b1ba8-kube-api-access-xv5d7\") pod \"payload-processing-e2e-shared-b-w6-4dcdbb-6bcbbcccc6-sbssx\" (UID: \"5c0c2a4a-8305-4fc2-a440-56d6725b1ba8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-4dcdbb-6bcbbcccc6-sbssx" Oct 30 17:37:52.989493 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:52.989441 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c0c2a4a-8305-4fc2-a440-56d6725b1ba8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-4dcdbb-6bcbbcccc6-sbssx\" (UID: \"5c0c2a4a-8305-4fc2-a440-56d6725b1ba8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-4dcdbb-6bcbbcccc6-sbssx" Oct 30 17:37:53.007773 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:53.003819 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c0c2a4a-8305-4fc2-a440-56d6725b1ba8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-4dcdbb-6bcbbcccc6-sbssx\" (UID: \"5c0c2a4a-8305-4fc2-a440-56d6725b1ba8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-4dcdbb-6bcbbcccc6-sbssx" Oct 30 17:37:53.007773 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:53.005354 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv5d7\" (UniqueName: \"kubernetes.io/projected/5c0c2a4a-8305-4fc2-a440-56d6725b1ba8-kube-api-access-xv5d7\") pod \"payload-processing-e2e-shared-b-w6-4dcdbb-6bcbbcccc6-sbssx\" (UID: \"5c0c2a4a-8305-4fc2-a440-56d6725b1ba8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-4dcdbb-6bcbbcccc6-sbssx" Oct 30 17:37:53.121953 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:53.118627 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv5d7\" (UniqueName: \"kubernetes.io/projected/5c0c2a4a-8305-4fc2-a440-56d6725b1ba8-kube-api-access-xv5d7\") pod \"payload-processing-e2e-shared-b-w6-4dcdbb-6bcbbcccc6-sbssx\" (UID: \"5c0c2a4a-8305-4fc2-a440-56d6725b1ba8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-4dcdbb-6bcbbcccc6-sbssx" Oct 30 17:37:53.173624 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:53.170391 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-4dcdbb-6bcbbcccc6-sbssx" Oct 30 17:37:53.416164 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:53.416125 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-4dcdbb-77d699b46c-qm59b"] Oct 30 17:37:53.532811 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:37:53.532771 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c0c2a4a_8305_4fc2_a440_56d6725b1ba8.slice/crio-a9388e616ec6decc0e48d19da3fb2529c7659c6cedcacdb10af5f1670f4f6f13 WatchSource:0}: Error finding container a9388e616ec6decc0e48d19da3fb2529c7659c6cedcacdb10af5f1670f4f6f13: Status 404 returned error can't find the container with id a9388e616ec6decc0e48d19da3fb2529c7659c6cedcacdb10af5f1670f4f6f13 Oct 30 17:37:53.533715 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:53.533688 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-4dcdbb-6bcbbcccc6-sbssx"] Oct 30 17:37:53.766590 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:53.766550 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-4dcdbb-6bcbbcccc6-sbssx" event={"ID":"5c0c2a4a-8305-4fc2-a440-56d6725b1ba8","Type":"ContainerStarted","Data":"2588799f55a52948f9f4338ac2bf5adb30bc6e5b0a1de4c0aa80f16dd5dccfa4"} Oct 30 17:37:53.766781 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:53.766612 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-4dcdbb-6bcbbcccc6-sbssx" Oct 30 17:37:53.766781 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:53.766660 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-4dcdbb-6bcbbcccc6-sbssx" event={"ID":"5c0c2a4a-8305-4fc2-a440-56d6725b1ba8","Type":"ContainerStarted","Data":"a9388e616ec6decc0e48d19da3fb2529c7659c6cedcacdb10af5f1670f4f6f13"} Oct 30 17:37:53.768267 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:53.768233 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-4dcdbb-77d699b46c-qm59b" event={"ID":"8dec2e67-e439-47cc-bf7b-b315dae10d2e","Type":"ContainerStarted","Data":"afec01a666dc208543de81adf6c7eb40a06822ca389eeb687794293aed55a571"} Oct 30 17:37:53.768401 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:53.768279 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-4dcdbb-77d699b46c-qm59b" event={"ID":"8dec2e67-e439-47cc-bf7b-b315dae10d2e","Type":"ContainerStarted","Data":"f831da814c0282e4eb839e6cc4e2b9b374269c2fa29f784522f0a1bf5e2cdf45"} Oct 30 17:37:53.768401 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:53.768323 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-4dcdbb-77d699b46c-qm59b" Oct 30 17:37:53.817393 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:53.817298 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-4dcdbb-6bcbbcccc6-sbssx" podStartSLOduration=1.8172862250000001 podStartE2EDuration="1.817286225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:37:53.814262715 +0000 UTC m=+1531.866362672" watchObservedRunningTime="2026-10-30 17:37:53.817286225 +0000 UTC m=+1531.869386354" Oct 30 17:37:53.858997 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:53.858945 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-4dcdbb-77d699b46c-qm59b" podStartSLOduration=1.8589308390000001 podStartE2EDuration="1.858930839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:37:53.855957226 +0000 UTC m=+1531.908057157" watchObservedRunningTime="2026-10-30 17:37:53.858930839 +0000 UTC m=+1531.911030770" Oct 30 17:37:54.366413 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:54.366373 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d1c973-5d95487bd7-vmdzr"] Oct 30 17:37:54.371404 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:54.371382 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1c973-5d95487bd7-vmdzr" Oct 30 17:37:54.389725 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:54.388875 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d1c973\"" Oct 30 17:37:54.389725 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:54.388895 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d1c973-dockercfg-xhdtl\"" Oct 30 17:37:54.390517 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:54.390496 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d1c973-5d95487bd7-vmdzr"] Oct 30 17:37:54.515905 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:54.501256 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kpdr\" (UniqueName: \"kubernetes.io/projected/743b47bf-ff68-4a25-bb33-785d836c1cc9-kube-api-access-8kpdr\") pod \"payload-pre-processing-e2e-api-b-d1c973-5d95487bd7-vmdzr\" (UID: \"743b47bf-ff68-4a25-bb33-785d836c1cc9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1c973-5d95487bd7-vmdzr" Oct 30 17:37:54.533439 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:54.533396 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/743b47bf-ff68-4a25-bb33-785d836c1cc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d1c973-5d95487bd7-vmdzr\" (UID: \"743b47bf-ff68-4a25-bb33-785d836c1cc9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1c973-5d95487bd7-vmdzr" Oct 30 17:37:54.689150 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:54.664927 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d1c973-5d4f7449db-8dh8t"] Oct 30 17:37:54.774948 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:54.766682 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/743b47bf-ff68-4a25-bb33-785d836c1cc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d1c973-5d95487bd7-vmdzr\" (UID: \"743b47bf-ff68-4a25-bb33-785d836c1cc9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1c973-5d95487bd7-vmdzr" Oct 30 17:37:54.774948 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:54.767128 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kpdr\" (UniqueName: \"kubernetes.io/projected/743b47bf-ff68-4a25-bb33-785d836c1cc9-kube-api-access-8kpdr\") pod \"payload-pre-processing-e2e-api-b-d1c973-5d95487bd7-vmdzr\" (UID: \"743b47bf-ff68-4a25-bb33-785d836c1cc9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1c973-5d95487bd7-vmdzr" Oct 30 17:37:54.886852 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:54.886812 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/743b47bf-ff68-4a25-bb33-785d836c1cc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d1c973-5d95487bd7-vmdzr\" (UID: \"743b47bf-ff68-4a25-bb33-785d836c1cc9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1c973-5d95487bd7-vmdzr" Oct 30 17:37:54.906431 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:54.906394 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d1c973-5d4f7449db-8dh8t" Oct 30 17:37:54.906962 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:54.906933 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d1c973-5d4f7449db-8dh8t"] Oct 30 17:37:55.013576 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:55.010182 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kpdr\" (UniqueName: \"kubernetes.io/projected/743b47bf-ff68-4a25-bb33-785d836c1cc9-kube-api-access-8kpdr\") pod \"payload-pre-processing-e2e-api-b-d1c973-5d95487bd7-vmdzr\" (UID: \"743b47bf-ff68-4a25-bb33-785d836c1cc9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1c973-5d95487bd7-vmdzr" Oct 30 17:37:55.013576 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:55.010642 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32b71d3c-848d-453a-8a75-5b3e0281050d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d1c973-5d4f7449db-8dh8t\" (UID: \"32b71d3c-848d-453a-8a75-5b3e0281050d\") " pod="openshift-ingress/payload-processing-e2e-api-b-d1c973-5d4f7449db-8dh8t" Oct 30 17:37:55.013576 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:55.010687 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5pf2\" (UniqueName: \"kubernetes.io/projected/32b71d3c-848d-453a-8a75-5b3e0281050d-kube-api-access-v5pf2\") pod \"payload-processing-e2e-api-b-d1c973-5d4f7449db-8dh8t\" (UID: \"32b71d3c-848d-453a-8a75-5b3e0281050d\") " pod="openshift-ingress/payload-processing-e2e-api-b-d1c973-5d4f7449db-8dh8t" Oct 30 17:37:55.112093 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:55.112064 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32b71d3c-848d-453a-8a75-5b3e0281050d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d1c973-5d4f7449db-8dh8t\" (UID: \"32b71d3c-848d-453a-8a75-5b3e0281050d\") " pod="openshift-ingress/payload-processing-e2e-api-b-d1c973-5d4f7449db-8dh8t" Oct 30 17:37:55.112278 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:55.112104 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5pf2\" (UniqueName: \"kubernetes.io/projected/32b71d3c-848d-453a-8a75-5b3e0281050d-kube-api-access-v5pf2\") pod \"payload-processing-e2e-api-b-d1c973-5d4f7449db-8dh8t\" (UID: \"32b71d3c-848d-453a-8a75-5b3e0281050d\") " pod="openshift-ingress/payload-processing-e2e-api-b-d1c973-5d4f7449db-8dh8t" Oct 30 17:37:55.112821 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:55.112798 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32b71d3c-848d-453a-8a75-5b3e0281050d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d1c973-5d4f7449db-8dh8t\" (UID: \"32b71d3c-848d-453a-8a75-5b3e0281050d\") " pod="openshift-ingress/payload-processing-e2e-api-b-d1c973-5d4f7449db-8dh8t" Oct 30 17:37:55.148616 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:55.148585 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5pf2\" (UniqueName: \"kubernetes.io/projected/32b71d3c-848d-453a-8a75-5b3e0281050d-kube-api-access-v5pf2\") pod \"payload-processing-e2e-api-b-d1c973-5d4f7449db-8dh8t\" (UID: \"32b71d3c-848d-453a-8a75-5b3e0281050d\") " pod="openshift-ingress/payload-processing-e2e-api-b-d1c973-5d4f7449db-8dh8t" Oct 30 17:37:55.242774 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:55.242745 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d1c973-5d4f7449db-8dh8t" Oct 30 17:37:55.289335 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:55.289252 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1c973-5d95487bd7-vmdzr" Oct 30 17:37:55.560989 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:37:55.560953 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32b71d3c_848d_453a_8a75_5b3e0281050d.slice/crio-f4d2b4d041d42fecc564cdcc70df89ad18695376eecc78cb75afe3251269c19a WatchSource:0}: Error finding container f4d2b4d041d42fecc564cdcc70df89ad18695376eecc78cb75afe3251269c19a: Status 404 returned error can't find the container with id f4d2b4d041d42fecc564cdcc70df89ad18695376eecc78cb75afe3251269c19a Oct 30 17:37:55.561407 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:55.561122 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d1c973-5d4f7449db-8dh8t"] Oct 30 17:37:55.641801 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:37:55.641771 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod743b47bf_ff68_4a25_bb33_785d836c1cc9.slice/crio-9e63c2a72ae275d37a4cd1fca650f4260be29464880fa5592c2a126a25018979 WatchSource:0}: Error finding container 9e63c2a72ae275d37a4cd1fca650f4260be29464880fa5592c2a126a25018979: Status 404 returned error can't find the container with id 9e63c2a72ae275d37a4cd1fca650f4260be29464880fa5592c2a126a25018979 Oct 30 17:37:55.643710 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:55.643684 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d1c973-5d95487bd7-vmdzr"] Oct 30 17:37:55.907272 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:55.907238 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1c973-5d95487bd7-vmdzr" event={"ID":"743b47bf-ff68-4a25-bb33-785d836c1cc9","Type":"ContainerStarted","Data":"82888a930a858928e9f0489be629e964206ce2fdb9d19bd258f4354f867f1f66"} Oct 30 17:37:55.907490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:55.907317 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1c973-5d95487bd7-vmdzr" event={"ID":"743b47bf-ff68-4a25-bb33-785d836c1cc9","Type":"ContainerStarted","Data":"9e63c2a72ae275d37a4cd1fca650f4260be29464880fa5592c2a126a25018979"} Oct 30 17:37:55.907490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:55.907370 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1c973-5d95487bd7-vmdzr" Oct 30 17:37:55.909071 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:55.909043 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d1c973-5d4f7449db-8dh8t" event={"ID":"32b71d3c-848d-453a-8a75-5b3e0281050d","Type":"ContainerStarted","Data":"f67e2fc7dcd3992c193c8c5616a6f952fe9ef47a55e8a062080c2e3268f885a1"} Oct 30 17:37:55.909185 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:55.909088 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d1c973-5d4f7449db-8dh8t" event={"ID":"32b71d3c-848d-453a-8a75-5b3e0281050d","Type":"ContainerStarted","Data":"f4d2b4d041d42fecc564cdcc70df89ad18695376eecc78cb75afe3251269c19a"} Oct 30 17:37:55.909185 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:55.909117 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d1c973-5d4f7449db-8dh8t" Oct 30 17:37:55.954688 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:55.954644 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1c973-5d95487bd7-vmdzr" podStartSLOduration=1.954630904 podStartE2EDuration="1.954630904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:37:55.95274275 +0000 UTC m=+1534.004842682" watchObservedRunningTime="2026-10-30 17:37:55.954630904 +0000 UTC m=+1534.006730880" Oct 30 17:37:56.000102 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:37:56.000048 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d1c973-5d4f7449db-8dh8t" podStartSLOduration=2.000030234 podStartE2EDuration="2.000030234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:37:55.995780399 +0000 UTC m=+1534.047880358" watchObservedRunningTime="2026-10-30 17:37:56.000030234 +0000 UTC m=+1534.052130193" Oct 30 17:38:04.909013 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:04.908984 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-4dcdbb-77d699b46c-qm59b" Oct 30 17:38:04.909479 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:04.909061 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-4dcdbb-6bcbbcccc6-sbssx" Oct 30 17:38:06.064615 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:38:06.064019 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:38:06.920463 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:06.920431 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d1c973-5d4f7449db-8dh8t" Oct 30 17:38:06.920921 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:06.920899 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1c973-5d95487bd7-vmdzr" Oct 30 17:38:07.052009 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:07.051759 4965 generic.go:356] "Generic (PLEG): container finished" podID="551401b3-f7a0-4b52-afac-f68219c56970" containerID="ff1c89d80bdcd2394bd9266df2d3fa4dc1d7f322448d97b16ae1a5b52e64c535" exitCode=0 Oct 30 17:38:07.052009 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:07.052112 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" event={"ID":"551401b3-f7a0-4b52-afac-f68219c56970","Type":"ContainerDied","Data":"ff1c89d80bdcd2394bd9266df2d3fa4dc1d7f322448d97b16ae1a5b52e64c535"} Oct 30 17:38:07.052935 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:07.052912 4965 scope.go:122] "RemoveContainer" containerID="ff1c89d80bdcd2394bd9266df2d3fa4dc1d7f322448d97b16ae1a5b52e64c535" Oct 30 17:38:07.055311 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:07.055272 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:38:07.055435 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:07.055353 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:38:07.055435 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:07.055392 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:38:08.087154 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:08.087114 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" event={"ID":"551401b3-f7a0-4b52-afac-f68219c56970","Type":"ContainerStarted","Data":"751c9b025e13eda4f301b78135ab90577126992f0a72c5f5707b4d8ff7375b0b"} Oct 30 17:38:08.140049 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:08.140019 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:38:08.140303 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:08.140281 4965 patch_prober.go:29] interesting pod/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 17:38:08.140355 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:08.140329 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" podUID="551401b3-f7a0-4b52-afac-f68219c56970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 17:38:09.140598 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:09.140558 4965 patch_prober.go:29] interesting pod/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 17:38:09.141059 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:09.140616 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" podUID="551401b3-f7a0-4b52-afac-f68219c56970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 17:38:10.140748 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:10.140712 4965 patch_prober.go:29] interesting pod/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 17:38:10.141285 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:10.140772 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" podUID="551401b3-f7a0-4b52-afac-f68219c56970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 17:38:11.139935 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:11.139858 4965 patch_prober.go:29] interesting pod/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 17:38:11.140142 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:11.139951 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" podUID="551401b3-f7a0-4b52-afac-f68219c56970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 17:38:12.140590 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:12.140556 4965 patch_prober.go:29] interesting pod/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 17:38:12.141100 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:12.140623 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" podUID="551401b3-f7a0-4b52-afac-f68219c56970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 17:38:13.140808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:13.140770 4965 patch_prober.go:29] interesting pod/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 17:38:13.141344 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:13.140859 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" podUID="551401b3-f7a0-4b52-afac-f68219c56970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 17:38:14.145153 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:14.145121 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:38:14.145543 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:14.145526 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:38:14.146391 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:14.146372 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 17:38:15.558847 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:15.558806 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-phmnv"] Oct 30 17:38:20.560062 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:20.559943 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-phmnv" podUID="ab8cfab6-dffc-4922-9106-12ca7f613b5b" containerName="payload-processing" containerID="cri-o://848305e4e194e9b95c117065ac7a93df5f5cc7c4b936c1787c292e233d190cab" gracePeriod=25 Oct 30 17:38:22.080717 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:22.080682 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc"] Oct 30 17:38:22.085824 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:22.085801 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc" Oct 30 17:38:22.126632 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:22.126598 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc"] Oct 30 17:38:22.159742 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:22.159716 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d1c973-serving-cert\"" Oct 30 17:38:22.231075 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:22.230744 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2zg7\" (UniqueName: \"kubernetes.io/projected/936c85c5-969e-47f9-b58f-57af5a0d36d9-kube-api-access-k2zg7\") pod \"maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc\" (UID: \"936c85c5-969e-47f9-b58f-57af5a0d36d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc" Oct 30 17:38:22.231075 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:22.230823 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/936c85c5-969e-47f9-b58f-57af5a0d36d9-metrics-certs\") pod \"maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc\" (UID: \"936c85c5-969e-47f9-b58f-57af5a0d36d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc" Oct 30 17:38:22.231075 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:22.230959 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/936c85c5-969e-47f9-b58f-57af5a0d36d9-openshift-service-ca\") pod \"maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc\" (UID: \"936c85c5-969e-47f9-b58f-57af5a0d36d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc" Oct 30 17:38:22.231323 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:22.231107 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/936c85c5-969e-47f9-b58f-57af5a0d36d9-maas-api-tls\") pod \"maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc\" (UID: \"936c85c5-969e-47f9-b58f-57af5a0d36d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc" Oct 30 17:38:22.348932 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:22.346369 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2zg7\" (UniqueName: \"kubernetes.io/projected/936c85c5-969e-47f9-b58f-57af5a0d36d9-kube-api-access-k2zg7\") pod \"maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc\" (UID: \"936c85c5-969e-47f9-b58f-57af5a0d36d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc" Oct 30 17:38:22.348932 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:22.346458 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/936c85c5-969e-47f9-b58f-57af5a0d36d9-metrics-certs\") pod \"maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc\" (UID: \"936c85c5-969e-47f9-b58f-57af5a0d36d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc" Oct 30 17:38:22.348932 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:22.347540 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/936c85c5-969e-47f9-b58f-57af5a0d36d9-openshift-service-ca\") pod \"maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc\" (UID: \"936c85c5-969e-47f9-b58f-57af5a0d36d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc" Oct 30 17:38:22.348932 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:22.347640 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/936c85c5-969e-47f9-b58f-57af5a0d36d9-maas-api-tls\") pod \"maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc\" (UID: \"936c85c5-969e-47f9-b58f-57af5a0d36d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc" Oct 30 17:38:22.348932 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:22.348898 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/936c85c5-969e-47f9-b58f-57af5a0d36d9-openshift-service-ca\") pod \"maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc\" (UID: \"936c85c5-969e-47f9-b58f-57af5a0d36d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc" Oct 30 17:38:22.405262 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:22.389928 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/936c85c5-969e-47f9-b58f-57af5a0d36d9-metrics-certs\") pod \"maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc\" (UID: \"936c85c5-969e-47f9-b58f-57af5a0d36d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc" Oct 30 17:38:22.405262 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:22.390649 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2zg7\" (UniqueName: \"kubernetes.io/projected/936c85c5-969e-47f9-b58f-57af5a0d36d9-kube-api-access-k2zg7\") pod \"maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc\" (UID: \"936c85c5-969e-47f9-b58f-57af5a0d36d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc" Oct 30 17:38:22.410671 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:22.410643 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/936c85c5-969e-47f9-b58f-57af5a0d36d9-maas-api-tls\") pod \"maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc\" (UID: \"936c85c5-969e-47f9-b58f-57af5a0d36d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc" Oct 30 17:38:22.411279 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:22.411249 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc" Oct 30 17:38:22.780262 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:38:22.780231 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod936c85c5_969e_47f9_b58f_57af5a0d36d9.slice/crio-3d1c60f491f55f4059dd3e31deff7f44a207298db4ef40637e77ec1c2c82b57a WatchSource:0}: Error finding container 3d1c60f491f55f4059dd3e31deff7f44a207298db4ef40637e77ec1c2c82b57a: Status 404 returned error can't find the container with id 3d1c60f491f55f4059dd3e31deff7f44a207298db4ef40637e77ec1c2c82b57a Oct 30 17:38:22.781036 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:22.781014 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc"] Oct 30 17:38:23.403260 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:23.403093 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc" event={"ID":"936c85c5-969e-47f9-b58f-57af5a0d36d9","Type":"ContainerStarted","Data":"ab74cd7b3135112ba3e8df91ec3c42c226866cba3eefb4834eeb0d77d7b682a1"} Oct 30 17:38:23.403889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:23.403845 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc" event={"ID":"936c85c5-969e-47f9-b58f-57af5a0d36d9","Type":"ContainerStarted","Data":"3d1c60f491f55f4059dd3e31deff7f44a207298db4ef40637e77ec1c2c82b57a"} Oct 30 17:38:23.409618 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:23.405500 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc" Oct 30 17:38:23.416066 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:23.415950 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-phmnv" podUID="ab8cfab6-dffc-4922-9106-12ca7f613b5b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:38:23.477881 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:23.474929 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc" podStartSLOduration=1.13596653 podStartE2EDuration="1.474908515s" totalImagesPullingTime="338.941985ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:38:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:38:23.463623192 +0000 UTC m=+1561.515723124" watchObservedRunningTime="2026-10-30 17:38:23.474908515 +0000 UTC m=+1561.527008446" Oct 30 17:38:24.419963 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:24.419930 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc"] Oct 30 17:38:24.420473 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:24.420114 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" podUID="53bbece8-96fa-412b-b259-fece0b49a345" containerName="istio-proxy" containerID="cri-o://5b656994fd096c028872c0c2449bdedaee006c49528e97e39dc2fa758ae89a65" gracePeriod=30 Oct 30 17:38:24.575423 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:24.575387 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm"] Oct 30 17:38:24.575664 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:24.575637 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" podUID="d493c105-7ba4-4626-b438-2224db37a905" containerName="istio-proxy" containerID="cri-o://47ea6ede04b24658360eb73bdc317de52c07c1353ff256597cb85d778898878f" gracePeriod=30 Oct 30 17:38:24.580206 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:24.580177 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx"] Oct 30 17:38:24.580546 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:24.580403 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" podUID="3f4d48ca-a065-4a13-89a7-efcc2301046f" containerName="istio-proxy" containerID="cri-o://31f5b061eb9dd73509ab603cb30a25db20fc90ad4a8f6bff85f2e069b2332034" gracePeriod=30 Oct 30 17:38:24.584827 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:24.584801 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw"] Oct 30 17:38:24.585198 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:24.585169 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" podUID="5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4" containerName="istio-proxy" containerID="cri-o://41e8ee7eab6820251f5a672d3ab50d3f46b6ff5353cd4feae259e566d8ed2ea1" gracePeriod=30 Oct 30 17:38:27.192887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:27.185965 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 17:38:27.212886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:27.211490 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" podUID="d174a74f-7205-4020-94b0-e30652c9b721" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Oct 30 17:38:27.545652 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:27.545321 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:38:27.953887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:27.950965 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 17:38:28.057887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:28.056974 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 17:38:28.078885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:28.077171 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:38:28.255172 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:28.254011 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:38:28.287888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:28.282329 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:38:28.319884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:28.315449 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" podUID="d174a74f-7205-4020-94b0-e30652c9b721" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:38:28.334924 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:28.333146 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" podUID="7715de9f-65c1-44fb-b83a-692ec07b30a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": context deadline exceeded" Oct 30 17:38:28.423724 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:28.423679 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:38:28.423952 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:28.423761 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" Oct 30 17:38:28.436364 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:28.436327 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" Oct 30 17:38:28.436596 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:28.436580 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 17:38:29.834421 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.834392 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:38:29.885817 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.885787 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:38:29.963947 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.963909 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53bbece8-96fa-412b-b259-fece0b49a345-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53bbece8-96fa-412b-b259-fece0b49a345-istio-podinfo\") pod \"53bbece8-96fa-412b-b259-fece0b49a345\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " Oct 30 17:38:29.964180 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.963979 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-workload-socket\") pod \"3f4d48ca-a065-4a13-89a7-efcc2301046f\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " Oct 30 17:38:29.964180 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.964018 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53bbece8-96fa-412b-b259-fece0b49a345-kube-api-access-5dr77\" (UniqueName: \"kubernetes.io/projected/53bbece8-96fa-412b-b259-fece0b49a345-kube-api-access-5dr77\") pod \"53bbece8-96fa-412b-b259-fece0b49a345\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " Oct 30 17:38:29.964180 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.964081 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-workload-socket\") pod \"53bbece8-96fa-412b-b259-fece0b49a345\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " Oct 30 17:38:29.964180 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.964130 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53bbece8-96fa-412b-b259-fece0b49a345-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53bbece8-96fa-412b-b259-fece0b49a345-istiod-ca-cert\") pod \"53bbece8-96fa-412b-b259-fece0b49a345\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " Oct 30 17:38:29.964437 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.964221 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53bbece8-96fa-412b-b259-fece0b49a345-istio-token\" (UniqueName: \"kubernetes.io/projected/53bbece8-96fa-412b-b259-fece0b49a345-istio-token\") pod \"53bbece8-96fa-412b-b259-fece0b49a345\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " Oct 30 17:38:29.964437 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.964277 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-workload-certs\") pod \"53bbece8-96fa-412b-b259-fece0b49a345\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " Oct 30 17:38:29.964437 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.964315 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-workload-certs\") pod \"3f4d48ca-a065-4a13-89a7-efcc2301046f\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " Oct 30 17:38:29.964437 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.964356 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-istio-data\") pod \"53bbece8-96fa-412b-b259-fece0b49a345\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " Oct 30 17:38:29.964437 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.964387 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-credential-socket\") pod \"53bbece8-96fa-412b-b259-fece0b49a345\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " Oct 30 17:38:29.964696 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.964449 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f4d48ca-a065-4a13-89a7-efcc2301046f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4d48ca-a065-4a13-89a7-efcc2301046f-istiod-ca-cert\") pod \"3f4d48ca-a065-4a13-89a7-efcc2301046f\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " Oct 30 17:38:29.964696 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.964539 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-istio-data\") pod \"3f4d48ca-a065-4a13-89a7-efcc2301046f\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " Oct 30 17:38:29.964696 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.964597 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-credential-socket\") pod \"3f4d48ca-a065-4a13-89a7-efcc2301046f\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " Oct 30 17:38:29.964696 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.964628 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f4d48ca-a065-4a13-89a7-efcc2301046f-kube-api-access-dh9nl\" (UniqueName: \"kubernetes.io/projected/3f4d48ca-a065-4a13-89a7-efcc2301046f-kube-api-access-dh9nl\") pod \"3f4d48ca-a065-4a13-89a7-efcc2301046f\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " Oct 30 17:38:29.964696 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.964681 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f4d48ca-a065-4a13-89a7-efcc2301046f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4d48ca-a065-4a13-89a7-efcc2301046f-istio-podinfo\") pod \"3f4d48ca-a065-4a13-89a7-efcc2301046f\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " Oct 30 17:38:29.965017 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.964755 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-istio-envoy\") pod \"53bbece8-96fa-412b-b259-fece0b49a345\" (UID: \"53bbece8-96fa-412b-b259-fece0b49a345\") " Oct 30 17:38:29.965017 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.964855 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f4d48ca-a065-4a13-89a7-efcc2301046f-istio-token\" (UniqueName: \"kubernetes.io/projected/3f4d48ca-a065-4a13-89a7-efcc2301046f-istio-token\") pod \"3f4d48ca-a065-4a13-89a7-efcc2301046f\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " Oct 30 17:38:29.965017 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.964918 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-istio-envoy\") pod \"3f4d48ca-a065-4a13-89a7-efcc2301046f\" (UID: \"3f4d48ca-a065-4a13-89a7-efcc2301046f\") " Oct 30 17:38:29.967108 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.966111 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-istio-data" pod "53bbece8-96fa-412b-b259-fece0b49a345" (UID: "53bbece8-96fa-412b-b259-fece0b49a345"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:29.967745 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.967362 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-credential-socket" pod "3f4d48ca-a065-4a13-89a7-efcc2301046f" (UID: "3f4d48ca-a065-4a13-89a7-efcc2301046f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:29.967819 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.967770 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f4d48ca-a065-4a13-89a7-efcc2301046f-istiod-ca-cert" pod "3f4d48ca-a065-4a13-89a7-efcc2301046f" (UID: "3f4d48ca-a065-4a13-89a7-efcc2301046f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:38:29.968347 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.968317 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-credential-socket" pod "53bbece8-96fa-412b-b259-fece0b49a345" (UID: "53bbece8-96fa-412b-b259-fece0b49a345"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:29.968467 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.968439 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-workload-socket" pod "3f4d48ca-a065-4a13-89a7-efcc2301046f" (UID: "3f4d48ca-a065-4a13-89a7-efcc2301046f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:29.969053 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.968977 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-workload-socket" pod "53bbece8-96fa-412b-b259-fece0b49a345" (UID: "53bbece8-96fa-412b-b259-fece0b49a345"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:29.969161 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.969089 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-istio-data" pod "3f4d48ca-a065-4a13-89a7-efcc2301046f" (UID: "3f4d48ca-a065-4a13-89a7-efcc2301046f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:29.969663 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.969637 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:29.969760 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.969669 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:29.969760 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.969685 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:29.969760 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.969699 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:29.969760 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.969717 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4d48ca-a065-4a13-89a7-efcc2301046f-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:29.969760 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.969731 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:29.969760 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.969745 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:29.971150 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.970602 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-workload-certs" pod "53bbece8-96fa-412b-b259-fece0b49a345" (UID: "53bbece8-96fa-412b-b259-fece0b49a345"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:29.971414 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.971352 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53bbece8-96fa-412b-b259-fece0b49a345-istiod-ca-cert" pod "53bbece8-96fa-412b-b259-fece0b49a345" (UID: "53bbece8-96fa-412b-b259-fece0b49a345"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:38:29.972108 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.971832 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-workload-certs" pod "3f4d48ca-a065-4a13-89a7-efcc2301046f" (UID: "3f4d48ca-a065-4a13-89a7-efcc2301046f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:29.980111 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.980084 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:38:29.985203 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.985158 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-istio-envoy" pod "53bbece8-96fa-412b-b259-fece0b49a345" (UID: "53bbece8-96fa-412b-b259-fece0b49a345"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:29.985466 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.985309 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f4d48ca-a065-4a13-89a7-efcc2301046f-istio-token" pod "3f4d48ca-a065-4a13-89a7-efcc2301046f" (UID: "3f4d48ca-a065-4a13-89a7-efcc2301046f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:29.985466 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.985428 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-istio-envoy" pod "3f4d48ca-a065-4a13-89a7-efcc2301046f" (UID: "3f4d48ca-a065-4a13-89a7-efcc2301046f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:29.985768 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.985741 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f4d48ca-a065-4a13-89a7-efcc2301046f-istio-podinfo" pod "3f4d48ca-a065-4a13-89a7-efcc2301046f" (UID: "3f4d48ca-a065-4a13-89a7-efcc2301046f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:38:29.986080 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.986042 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f4d48ca-a065-4a13-89a7-efcc2301046f-kube-api-access-dh9nl" pod "3f4d48ca-a065-4a13-89a7-efcc2301046f" (UID: "3f4d48ca-a065-4a13-89a7-efcc2301046f"). InnerVolumeSpecName "kube-api-access-dh9nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:29.986365 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.986340 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53bbece8-96fa-412b-b259-fece0b49a345-istio-podinfo" pod "53bbece8-96fa-412b-b259-fece0b49a345" (UID: "53bbece8-96fa-412b-b259-fece0b49a345"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:38:29.986527 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.986500 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53bbece8-96fa-412b-b259-fece0b49a345-kube-api-access-5dr77" pod "53bbece8-96fa-412b-b259-fece0b49a345" (UID: "53bbece8-96fa-412b-b259-fece0b49a345"). InnerVolumeSpecName "kube-api-access-5dr77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:29.989342 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:29.989316 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53bbece8-96fa-412b-b259-fece0b49a345-istio-token" pod "53bbece8-96fa-412b-b259-fece0b49a345" (UID: "53bbece8-96fa-412b-b259-fece0b49a345"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:30.038550 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.038519 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:38:30.070565 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.070534 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d493c105-7ba4-4626-b438-2224db37a905-istio-token\" (UniqueName: \"kubernetes.io/projected/d493c105-7ba4-4626-b438-2224db37a905-istio-token\") pod \"d493c105-7ba4-4626-b438-2224db37a905\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " Oct 30 17:38:30.070734 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.070582 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-workload-socket\") pod \"d493c105-7ba4-4626-b438-2224db37a905\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " Oct 30 17:38:30.070894 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.070856 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-workload-certs\") pod \"d493c105-7ba4-4626-b438-2224db37a905\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " Oct 30 17:38:30.071227 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.071035 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-workload-socket" pod "d493c105-7ba4-4626-b438-2224db37a905" (UID: "d493c105-7ba4-4626-b438-2224db37a905"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:30.071322 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.071130 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-workload-certs" pod "d493c105-7ba4-4626-b438-2224db37a905" (UID: "d493c105-7ba4-4626-b438-2224db37a905"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:30.071322 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.071208 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-credential-socket\") pod \"d493c105-7ba4-4626-b438-2224db37a905\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " Oct 30 17:38:30.071322 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.071282 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d493c105-7ba4-4626-b438-2224db37a905-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d493c105-7ba4-4626-b438-2224db37a905-istiod-ca-cert\") pod \"d493c105-7ba4-4626-b438-2224db37a905\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " Oct 30 17:38:30.071322 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.071312 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-istio-data\") pod \"d493c105-7ba4-4626-b438-2224db37a905\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " Oct 30 17:38:30.071527 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.071347 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d493c105-7ba4-4626-b438-2224db37a905-kube-api-access-vxmbb\" (UniqueName: \"kubernetes.io/projected/d493c105-7ba4-4626-b438-2224db37a905-kube-api-access-vxmbb\") pod \"d493c105-7ba4-4626-b438-2224db37a905\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " Oct 30 17:38:30.071527 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.071406 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-credential-socket" pod "d493c105-7ba4-4626-b438-2224db37a905" (UID: "d493c105-7ba4-4626-b438-2224db37a905"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:30.071527 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.071490 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-istio-envoy\") pod \"d493c105-7ba4-4626-b438-2224db37a905\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " Oct 30 17:38:30.071694 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.071549 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d493c105-7ba4-4626-b438-2224db37a905-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d493c105-7ba4-4626-b438-2224db37a905-istio-podinfo\") pod \"d493c105-7ba4-4626-b438-2224db37a905\" (UID: \"d493c105-7ba4-4626-b438-2224db37a905\") " Oct 30 17:38:30.071694 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.071615 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-istio-data" pod "d493c105-7ba4-4626-b438-2224db37a905" (UID: "d493c105-7ba4-4626-b438-2224db37a905"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:30.071803 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.071697 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d493c105-7ba4-4626-b438-2224db37a905-istiod-ca-cert" pod "d493c105-7ba4-4626-b438-2224db37a905" (UID: "d493c105-7ba4-4626-b438-2224db37a905"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:38:30.072433 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.072409 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53bbece8-96fa-412b-b259-fece0b49a345-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.072532 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.072439 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.072532 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.072460 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53bbece8-96fa-412b-b259-fece0b49a345-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.072532 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.072473 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.072532 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.072486 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.072532 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.072498 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.072532 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.072512 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d493c105-7ba4-4626-b438-2224db37a905-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.072532 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.072523 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.072532 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.072534 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh9nl\" (UniqueName: \"kubernetes.io/projected/3f4d48ca-a065-4a13-89a7-efcc2301046f-kube-api-access-dh9nl\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.072980 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.072548 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4d48ca-a065-4a13-89a7-efcc2301046f-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.072980 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.072563 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53bbece8-96fa-412b-b259-fece0b49a345-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.072980 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.072576 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.072980 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.072588 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f4d48ca-a065-4a13-89a7-efcc2301046f-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.072980 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.072601 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4d48ca-a065-4a13-89a7-efcc2301046f-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.072980 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.072614 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53bbece8-96fa-412b-b259-fece0b49a345-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.072980 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.072626 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dr77\" (UniqueName: \"kubernetes.io/projected/53bbece8-96fa-412b-b259-fece0b49a345-kube-api-access-5dr77\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.073710 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.073677 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d493c105-7ba4-4626-b438-2224db37a905-istio-token" pod "d493c105-7ba4-4626-b438-2224db37a905" (UID: "d493c105-7ba4-4626-b438-2224db37a905"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:30.074516 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.074489 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-istio-envoy" pod "d493c105-7ba4-4626-b438-2224db37a905" (UID: "d493c105-7ba4-4626-b438-2224db37a905"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:30.074836 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.074814 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d493c105-7ba4-4626-b438-2224db37a905-istio-podinfo" pod "d493c105-7ba4-4626-b438-2224db37a905" (UID: "d493c105-7ba4-4626-b438-2224db37a905"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:38:30.075058 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.075042 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d493c105-7ba4-4626-b438-2224db37a905-kube-api-access-vxmbb" pod "d493c105-7ba4-4626-b438-2224db37a905" (UID: "d493c105-7ba4-4626-b438-2224db37a905"). InnerVolumeSpecName "kube-api-access-vxmbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:30.173686 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.173657 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-credential-socket\") pod \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " Oct 30 17:38:30.173846 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.173746 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istiod-ca-cert\") pod \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " Oct 30 17:38:30.173846 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.173792 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-workload-socket\") pod \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " Oct 30 17:38:30.173846 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.173828 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istio-data\") pod \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " Oct 30 17:38:30.174015 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.173856 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-kube-api-access-r7l64\" (UniqueName: \"kubernetes.io/projected/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-kube-api-access-r7l64\") pod \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " Oct 30 17:38:30.174015 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.173957 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-credential-socket" pod "5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4" (UID: "5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:30.174015 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.173987 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istio-token\" (UniqueName: \"kubernetes.io/projected/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istio-token\") pod \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " Oct 30 17:38:30.174237 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.174034 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istio-podinfo\") pod \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " Oct 30 17:38:30.174237 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.174090 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istio-envoy\") pod \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " Oct 30 17:38:30.174237 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.174124 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-workload-socket" pod "5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4" (UID: "5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:30.174237 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.174127 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-workload-certs\") pod \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\" (UID: \"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4\") " Oct 30 17:38:30.174237 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.174184 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istio-data" pod "5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4" (UID: "5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:30.174504 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.174236 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istiod-ca-cert" pod "5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4" (UID: "5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:38:30.174504 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.174399 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-workload-certs" pod "5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4" (UID: "5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:30.175188 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.175166 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.175188 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.175190 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d493c105-7ba4-4626-b438-2224db37a905-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.175352 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.175203 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.175352 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.175215 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d493c105-7ba4-4626-b438-2224db37a905-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.175352 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.175227 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.175352 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.175240 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d493c105-7ba4-4626-b438-2224db37a905-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.175352 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.175252 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.175352 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.175263 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.175352 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.175276 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxmbb\" (UniqueName: \"kubernetes.io/projected/d493c105-7ba4-4626-b438-2224db37a905-kube-api-access-vxmbb\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.176383 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.176362 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istio-podinfo" pod "5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4" (UID: "5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:38:30.176487 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.176469 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istio-envoy" pod "5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4" (UID: "5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:30.176703 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.176685 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istio-token" pod "5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4" (UID: "5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:30.176754 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.176715 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-kube-api-access-r7l64" pod "5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4" (UID: "5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4"). InnerVolumeSpecName "kube-api-access-r7l64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:30.277447 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.277275 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7l64\" (UniqueName: \"kubernetes.io/projected/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-kube-api-access-r7l64\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.277447 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.277315 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.277447 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.277329 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.277447 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.277342 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.427500 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.427472 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc" Oct 30 17:38:30.534991 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.534888 4965 generic.go:356] "Generic (PLEG): container finished" podID="53bbece8-96fa-412b-b259-fece0b49a345" containerID="5b656994fd096c028872c0c2449bdedaee006c49528e97e39dc2fa758ae89a65" exitCode=0 Oct 30 17:38:30.534991 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.534944 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" Oct 30 17:38:30.535234 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.534981 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" event={"ID":"53bbece8-96fa-412b-b259-fece0b49a345","Type":"ContainerDied","Data":"5b656994fd096c028872c0c2449bdedaee006c49528e97e39dc2fa758ae89a65"} Oct 30 17:38:30.535501 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.535464 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc" event={"ID":"53bbece8-96fa-412b-b259-fece0b49a345","Type":"ContainerDied","Data":"9cc6d4c66ef839b1aa7ae3c9eb47ea4c8c92b4fd11b9552e39cd9bc94934901e"} Oct 30 17:38:30.535598 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.535525 4965 scope.go:122] "RemoveContainer" containerID="5b656994fd096c028872c0c2449bdedaee006c49528e97e39dc2fa758ae89a65" Oct 30 17:38:30.537989 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.537964 4965 generic.go:356] "Generic (PLEG): container finished" podID="d493c105-7ba4-4626-b438-2224db37a905" containerID="47ea6ede04b24658360eb73bdc317de52c07c1353ff256597cb85d778898878f" exitCode=0 Oct 30 17:38:30.538249 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.538056 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" Oct 30 17:38:30.538921 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.538075 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" event={"ID":"d493c105-7ba4-4626-b438-2224db37a905","Type":"ContainerDied","Data":"47ea6ede04b24658360eb73bdc317de52c07c1353ff256597cb85d778898878f"} Oct 30 17:38:30.539095 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.539073 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm" event={"ID":"d493c105-7ba4-4626-b438-2224db37a905","Type":"ContainerDied","Data":"87d27bd37225802032b9c6b32a994c9dc4386368244cc9bbbf0e55a92f7042cd"} Oct 30 17:38:30.540805 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.540784 4965 generic.go:356] "Generic (PLEG): container finished" podID="3f4d48ca-a065-4a13-89a7-efcc2301046f" containerID="31f5b061eb9dd73509ab603cb30a25db20fc90ad4a8f6bff85f2e069b2332034" exitCode=0 Oct 30 17:38:30.541209 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.540992 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" event={"ID":"3f4d48ca-a065-4a13-89a7-efcc2301046f","Type":"ContainerDied","Data":"31f5b061eb9dd73509ab603cb30a25db20fc90ad4a8f6bff85f2e069b2332034"} Oct 30 17:38:30.541631 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.541022 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" Oct 30 17:38:30.541787 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.541763 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx" event={"ID":"3f4d48ca-a065-4a13-89a7-efcc2301046f","Type":"ContainerDied","Data":"84cd380b4683cef8dc9acf1a2589ccb1a1e24a341ebab21dcf48702c8ce0b8ac"} Oct 30 17:38:30.543243 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.543220 4965 generic.go:356] "Generic (PLEG): container finished" podID="5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4" containerID="41e8ee7eab6820251f5a672d3ab50d3f46b6ff5353cd4feae259e566d8ed2ea1" exitCode=0 Oct 30 17:38:30.543349 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.543264 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" Oct 30 17:38:30.543421 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.543402 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" event={"ID":"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4","Type":"ContainerDied","Data":"41e8ee7eab6820251f5a672d3ab50d3f46b6ff5353cd4feae259e566d8ed2ea1"} Oct 30 17:38:30.543489 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.543447 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw" event={"ID":"5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4","Type":"ContainerDied","Data":"c65f2e4f2602b684ec777f63a23b6c75ac6451fde2b1b2196a34eeeebe2b1eb4"} Oct 30 17:38:30.555395 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.555376 4965 scope.go:122] "RemoveContainer" containerID="5b656994fd096c028872c0c2449bdedaee006c49528e97e39dc2fa758ae89a65" Oct 30 17:38:30.555792 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:38:30.555761 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b656994fd096c028872c0c2449bdedaee006c49528e97e39dc2fa758ae89a65\": container with ID starting with 5b656994fd096c028872c0c2449bdedaee006c49528e97e39dc2fa758ae89a65 not found: ID does not exist" containerID="5b656994fd096c028872c0c2449bdedaee006c49528e97e39dc2fa758ae89a65" Oct 30 17:38:30.555941 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.555802 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b656994fd096c028872c0c2449bdedaee006c49528e97e39dc2fa758ae89a65"} err="failed to get container status \"5b656994fd096c028872c0c2449bdedaee006c49528e97e39dc2fa758ae89a65\": rpc error: code = NotFound desc = could not find container \"5b656994fd096c028872c0c2449bdedaee006c49528e97e39dc2fa758ae89a65\": container with ID starting with 5b656994fd096c028872c0c2449bdedaee006c49528e97e39dc2fa758ae89a65 not found: ID does not exist" Oct 30 17:38:30.555941 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.555828 4965 scope.go:122] "RemoveContainer" containerID="47ea6ede04b24658360eb73bdc317de52c07c1353ff256597cb85d778898878f" Oct 30 17:38:30.570362 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.570341 4965 scope.go:122] "RemoveContainer" containerID="47ea6ede04b24658360eb73bdc317de52c07c1353ff256597cb85d778898878f" Oct 30 17:38:30.571083 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:38:30.571058 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47ea6ede04b24658360eb73bdc317de52c07c1353ff256597cb85d778898878f\": container with ID starting with 47ea6ede04b24658360eb73bdc317de52c07c1353ff256597cb85d778898878f not found: ID does not exist" containerID="47ea6ede04b24658360eb73bdc317de52c07c1353ff256597cb85d778898878f" Oct 30 17:38:30.571171 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.571089 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47ea6ede04b24658360eb73bdc317de52c07c1353ff256597cb85d778898878f"} err="failed to get container status \"47ea6ede04b24658360eb73bdc317de52c07c1353ff256597cb85d778898878f\": rpc error: code = NotFound desc = could not find container \"47ea6ede04b24658360eb73bdc317de52c07c1353ff256597cb85d778898878f\": container with ID starting with 47ea6ede04b24658360eb73bdc317de52c07c1353ff256597cb85d778898878f not found: ID does not exist" Oct 30 17:38:30.571171 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.571110 4965 scope.go:122] "RemoveContainer" containerID="31f5b061eb9dd73509ab603cb30a25db20fc90ad4a8f6bff85f2e069b2332034" Oct 30 17:38:30.581500 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.581483 4965 scope.go:122] "RemoveContainer" containerID="31f5b061eb9dd73509ab603cb30a25db20fc90ad4a8f6bff85f2e069b2332034" Oct 30 17:38:30.581812 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:38:30.581793 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31f5b061eb9dd73509ab603cb30a25db20fc90ad4a8f6bff85f2e069b2332034\": container with ID starting with 31f5b061eb9dd73509ab603cb30a25db20fc90ad4a8f6bff85f2e069b2332034 not found: ID does not exist" containerID="31f5b061eb9dd73509ab603cb30a25db20fc90ad4a8f6bff85f2e069b2332034" Oct 30 17:38:30.581885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.581819 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31f5b061eb9dd73509ab603cb30a25db20fc90ad4a8f6bff85f2e069b2332034"} err="failed to get container status \"31f5b061eb9dd73509ab603cb30a25db20fc90ad4a8f6bff85f2e069b2332034\": rpc error: code = NotFound desc = could not find container \"31f5b061eb9dd73509ab603cb30a25db20fc90ad4a8f6bff85f2e069b2332034\": container with ID starting with 31f5b061eb9dd73509ab603cb30a25db20fc90ad4a8f6bff85f2e069b2332034 not found: ID does not exist" Oct 30 17:38:30.581885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.581837 4965 scope.go:122] "RemoveContainer" containerID="41e8ee7eab6820251f5a672d3ab50d3f46b6ff5353cd4feae259e566d8ed2ea1" Oct 30 17:38:30.608044 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.608012 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc"] Oct 30 17:38:30.612146 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.612114 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vqcgc"] Oct 30 17:38:30.613027 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.613002 4965 scope.go:122] "RemoveContainer" containerID="41e8ee7eab6820251f5a672d3ab50d3f46b6ff5353cd4feae259e566d8ed2ea1" Oct 30 17:38:30.613429 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:38:30.613403 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41e8ee7eab6820251f5a672d3ab50d3f46b6ff5353cd4feae259e566d8ed2ea1\": container with ID starting with 41e8ee7eab6820251f5a672d3ab50d3f46b6ff5353cd4feae259e566d8ed2ea1 not found: ID does not exist" containerID="41e8ee7eab6820251f5a672d3ab50d3f46b6ff5353cd4feae259e566d8ed2ea1" Oct 30 17:38:30.613514 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.613438 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41e8ee7eab6820251f5a672d3ab50d3f46b6ff5353cd4feae259e566d8ed2ea1"} err="failed to get container status \"41e8ee7eab6820251f5a672d3ab50d3f46b6ff5353cd4feae259e566d8ed2ea1\": rpc error: code = NotFound desc = could not find container \"41e8ee7eab6820251f5a672d3ab50d3f46b6ff5353cd4feae259e566d8ed2ea1\": container with ID starting with 41e8ee7eab6820251f5a672d3ab50d3f46b6ff5353cd4feae259e566d8ed2ea1 not found: ID does not exist" Oct 30 17:38:30.651533 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.651500 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm"] Oct 30 17:38:30.656444 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.656416 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-r85nm"] Oct 30 17:38:30.706965 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.706937 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx"] Oct 30 17:38:30.715499 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.715475 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-nq2xx"] Oct 30 17:38:30.763005 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.762956 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw"] Oct 30 17:38:30.768434 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:30.768406 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-6njpw"] Oct 30 17:38:32.101623 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:32.101592 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f4d48ca-a065-4a13-89a7-efcc2301046f" path="/var/lib/kubelet/pods/3f4d48ca-a065-4a13-89a7-efcc2301046f/volumes" Oct 30 17:38:32.102151 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:32.102129 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53bbece8-96fa-412b-b259-fece0b49a345" path="/var/lib/kubelet/pods/53bbece8-96fa-412b-b259-fece0b49a345/volumes" Oct 30 17:38:32.102788 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:32.102764 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4" path="/var/lib/kubelet/pods/5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4/volumes" Oct 30 17:38:32.103709 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:32.103689 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d493c105-7ba4-4626-b438-2224db37a905" path="/var/lib/kubelet/pods/d493c105-7ba4-4626-b438-2224db37a905/volumes" Oct 30 17:38:33.404050 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:33.403960 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-phmnv" podUID="ab8cfab6-dffc-4922-9106-12ca7f613b5b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:38:35.998389 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:38:35.998353 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:38:40.700709 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:40.700677 4965 generic.go:356] "Generic (PLEG): container finished" podID="ab8cfab6-dffc-4922-9106-12ca7f613b5b" containerID="848305e4e194e9b95c117065ac7a93df5f5cc7c4b936c1787c292e233d190cab" exitCode=0 Oct 30 17:38:40.701358 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:40.701323 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-phmnv" event={"ID":"ab8cfab6-dffc-4922-9106-12ca7f613b5b","Type":"ContainerDied","Data":"848305e4e194e9b95c117065ac7a93df5f5cc7c4b936c1787c292e233d190cab"} Oct 30 17:38:40.909934 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:40.909907 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-phmnv" Oct 30 17:38:40.992564 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:40.992532 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab8cfab6-dffc-4922-9106-12ca7f613b5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8cfab6-dffc-4922-9106-12ca7f613b5b-plugins-config-volume\") pod \"ab8cfab6-dffc-4922-9106-12ca7f613b5b\" (UID: \"ab8cfab6-dffc-4922-9106-12ca7f613b5b\") " Oct 30 17:38:40.992744 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:40.992575 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab8cfab6-dffc-4922-9106-12ca7f613b5b-kube-api-access-kftgh\" (UniqueName: \"kubernetes.io/projected/ab8cfab6-dffc-4922-9106-12ca7f613b5b-kube-api-access-kftgh\") pod \"ab8cfab6-dffc-4922-9106-12ca7f613b5b\" (UID: \"ab8cfab6-dffc-4922-9106-12ca7f613b5b\") " Oct 30 17:38:40.992976 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:40.992953 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab8cfab6-dffc-4922-9106-12ca7f613b5b-plugins-config-volume" pod "ab8cfab6-dffc-4922-9106-12ca7f613b5b" (UID: "ab8cfab6-dffc-4922-9106-12ca7f613b5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:38:40.993359 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:40.993337 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8cfab6-dffc-4922-9106-12ca7f613b5b-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:40.994947 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:40.994914 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab8cfab6-dffc-4922-9106-12ca7f613b5b-kube-api-access-kftgh" pod "ab8cfab6-dffc-4922-9106-12ca7f613b5b" (UID: "ab8cfab6-dffc-4922-9106-12ca7f613b5b"). InnerVolumeSpecName "kube-api-access-kftgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:41.095087 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:41.095054 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kftgh\" (UniqueName: \"kubernetes.io/projected/ab8cfab6-dffc-4922-9106-12ca7f613b5b-kube-api-access-kftgh\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:38:41.710911 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:41.710881 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-phmnv" Oct 30 17:38:41.711341 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:41.710879 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-phmnv" event={"ID":"ab8cfab6-dffc-4922-9106-12ca7f613b5b","Type":"ContainerDied","Data":"47c3f3542a4aade1c1096ba1198db84e4cb8fc084dafce45282505e8302802f7"} Oct 30 17:38:41.711341 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:41.711014 4965 scope.go:122] "RemoveContainer" containerID="848305e4e194e9b95c117065ac7a93df5f5cc7c4b936c1787c292e233d190cab" Oct 30 17:38:41.764928 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:41.764890 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-phmnv"] Oct 30 17:38:41.776508 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:41.776482 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-81e9336e-5f87896b8-phmnv"] Oct 30 17:38:42.101117 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:42.101086 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab8cfab6-dffc-4922-9106-12ca7f613b5b" path="/var/lib/kubelet/pods/ab8cfab6-dffc-4922-9106-12ca7f613b5b/volumes" Oct 30 17:38:54.610712 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:54.610674 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4"] Oct 30 17:38:54.611201 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:54.610922 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" podUID="f4953033-0273-4681-8d46-94128ab21e6c" containerName="istio-proxy" containerID="cri-o://aa9c503d179ee92a95cc9bc385e86b667d072ce5f0deb021da3cbf0531623bc2" gracePeriod=30 Oct 30 17:38:54.625784 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:54.625747 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq"] Oct 30 17:38:54.626095 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:54.626064 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" podUID="9f45f733-1397-4de5-81fd-b035baef4fb4" containerName="istio-proxy" containerID="cri-o://c55a6ba88b03e73fcc59835e1762a6237e0371c5e89c93dbe498707dda12691f" gracePeriod=30 Oct 30 17:38:54.631980 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:54.631958 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp"] Oct 30 17:38:54.632187 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:54.632168 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" podUID="5c868037-59d9-480a-87c1-f29251d35284" containerName="istio-proxy" containerID="cri-o://499923eb9585afcca4e0c93b287ca973a017b039f0c24a4f56aee25e2ce88d67" gracePeriod=30 Oct 30 17:38:58.287595 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:58.287544 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:38:58.868415 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:58.868364 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g"] Oct 30 17:38:58.870293 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:58.870244 4965 cpu_manager.go:414] "containerMap: removing container" podUID="3f4d48ca-a065-4a13-89a7-efcc2301046f" containerName="istio-proxy" Oct 30 17:38:58.870464 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:58.870450 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f4d48ca-a065-4a13-89a7-efcc2301046f" containerName="istio-proxy" Oct 30 17:38:58.870566 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:58.870556 4965 cpu_manager.go:414] "containerMap: removing container" podUID="d493c105-7ba4-4626-b438-2224db37a905" containerName="istio-proxy" Oct 30 17:38:58.870644 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:58.870634 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d493c105-7ba4-4626-b438-2224db37a905" containerName="istio-proxy" Oct 30 17:38:58.870780 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:58.870765 4965 cpu_manager.go:414] "containerMap: removing container" podUID="5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4" containerName="istio-proxy" Oct 30 17:38:58.870896 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:58.870880 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4" containerName="istio-proxy" Oct 30 17:38:58.871015 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:58.871003 4965 cpu_manager.go:414] "containerMap: removing container" podUID="ab8cfab6-dffc-4922-9106-12ca7f613b5b" containerName="payload-processing" Oct 30 17:38:58.871103 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:58.871093 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab8cfab6-dffc-4922-9106-12ca7f613b5b" containerName="payload-processing" Oct 30 17:38:58.871229 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:58.871218 4965 cpu_manager.go:414] "containerMap: removing container" podUID="53bbece8-96fa-412b-b259-fece0b49a345" containerName="istio-proxy" Oct 30 17:38:58.871305 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:58.871295 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53bbece8-96fa-412b-b259-fece0b49a345" containerName="istio-proxy" Oct 30 17:38:58.871563 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:58.871548 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bb6f0cf-bfbc-4b53-92f3-eac354a8d1a4" containerName="istio-proxy" Oct 30 17:38:58.871660 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:58.871650 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab8cfab6-dffc-4922-9106-12ca7f613b5b" containerName="payload-processing" Oct 30 17:38:58.871758 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:58.871747 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="53bbece8-96fa-412b-b259-fece0b49a345" containerName="istio-proxy" Oct 30 17:38:58.871857 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:58.871845 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f4d48ca-a065-4a13-89a7-efcc2301046f" containerName="istio-proxy" Oct 30 17:38:58.871994 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:58.871980 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="d493c105-7ba4-4626-b438-2224db37a905" containerName="istio-proxy" Oct 30 17:38:58.882282 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:58.882253 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:38:58.932287 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:58.925065 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c9bdd9-openshift-default-dockercfg-44mbg\"" Oct 30 17:38:58.989205 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:58.989170 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g"] Oct 30 17:38:59.075544 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:59.072446 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30e31aef-c00b-4418-b005-50b3689d2912-istiod-ca-cert\") pod \"e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:38:59.076632 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:59.076587 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76wnm\" (UniqueName: \"kubernetes.io/projected/30e31aef-c00b-4418-b005-50b3689d2912-kube-api-access-76wnm\") pod \"e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:38:59.080010 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:59.079922 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-credential-socket\") pod \"e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:38:59.080200 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:59.080182 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30e31aef-c00b-4418-b005-50b3689d2912-istio-podinfo\") pod \"e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:38:59.080335 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:59.080310 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-istio-envoy\") pod \"e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:38:59.080612 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:59.080601 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30e31aef-c00b-4418-b005-50b3689d2912-istio-token\") pod \"e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:38:59.080802 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:59.080789 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-istio-data\") pod \"e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:38:59.080963 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:59.080950 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-workload-socket\") pod \"e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:38:59.081106 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:59.081094 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-workload-certs\") pod \"e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:38:59.182375 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:59.182292 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30e31aef-c00b-4418-b005-50b3689d2912-istio-token\") pod \"e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:38:59.182558 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:59.182393 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-istio-data\") pod \"e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:38:59.182638 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:59.182548 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-workload-socket\") pod \"e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:38:59.182638 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:59.182610 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-workload-certs\") pod \"e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:38:59.182806 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:59.182784 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-istio-data\") pod \"e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:38:59.182846 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:59.182804 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30e31aef-c00b-4418-b005-50b3689d2912-istiod-ca-cert\") pod \"e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:38:59.182951 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:59.182908 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-workload-socket\") pod \"e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:38:59.182951 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:59.182916 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76wnm\" (UniqueName: \"kubernetes.io/projected/30e31aef-c00b-4418-b005-50b3689d2912-kube-api-access-76wnm\") pod \"e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:38:59.183062 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:59.182981 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-credential-socket\") pod \"e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:38:59.183062 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:59.183012 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-workload-certs\") pod \"e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:38:59.183062 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:59.183015 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30e31aef-c00b-4418-b005-50b3689d2912-istio-podinfo\") pod \"e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:38:59.183206 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:59.183094 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-istio-envoy\") pod \"e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:38:59.183517 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:59.183493 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-credential-socket\") pod \"e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:38:59.183517 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:59.183510 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30e31aef-c00b-4418-b005-50b3689d2912-istiod-ca-cert\") pod \"e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:38:59.185284 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:59.185264 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-istio-envoy\") pod \"e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:38:59.185539 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:59.185523 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30e31aef-c00b-4418-b005-50b3689d2912-istio-podinfo\") pod \"e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:38:59.205995 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:59.205967 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76wnm\" (UniqueName: \"kubernetes.io/projected/30e31aef-c00b-4418-b005-50b3689d2912-kube-api-access-76wnm\") pod \"e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:38:59.206125 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:59.206103 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30e31aef-c00b-4418-b005-50b3689d2912-istio-token\") pod \"e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:38:59.238743 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:59.238718 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:38:59.848367 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:59.834816 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g"] Oct 30 17:38:59.957917 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:59.896460 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:38:59.957917 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:59.896557 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:38:59.957917 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:38:59.896602 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:39:00.022276 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.022231 4965 generic.go:356] "Generic (PLEG): container finished" podID="9f45f733-1397-4de5-81fd-b035baef4fb4" containerID="c55a6ba88b03e73fcc59835e1762a6237e0371c5e89c93dbe498707dda12691f" exitCode=0 Oct 30 17:39:00.022417 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.022325 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" event={"ID":"9f45f733-1397-4de5-81fd-b035baef4fb4","Type":"ContainerDied","Data":"c55a6ba88b03e73fcc59835e1762a6237e0371c5e89c93dbe498707dda12691f"} Oct 30 17:39:00.024163 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.024133 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" event={"ID":"30e31aef-c00b-4418-b005-50b3689d2912","Type":"ContainerStarted","Data":"f7ee32cb2ebdf37b8bc4cb83924db1c64b4328b23bf3328e73161b3096e63078"} Oct 30 17:39:00.249422 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.249324 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:39:00.328704 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.328668 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f45f733-1397-4de5-81fd-b035baef4fb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f45f733-1397-4de5-81fd-b035baef4fb4-istiod-ca-cert\") pod \"9f45f733-1397-4de5-81fd-b035baef4fb4\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " Oct 30 17:39:00.329364 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.329339 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-istio-data\") pod \"9f45f733-1397-4de5-81fd-b035baef4fb4\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " Oct 30 17:39:00.330133 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.329262 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f45f733-1397-4de5-81fd-b035baef4fb4-istiod-ca-cert" pod "9f45f733-1397-4de5-81fd-b035baef4fb4" (UID: "9f45f733-1397-4de5-81fd-b035baef4fb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:39:00.330302 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.329807 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-istio-data" pod "9f45f733-1397-4de5-81fd-b035baef4fb4" (UID: "9f45f733-1397-4de5-81fd-b035baef4fb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:00.330513 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.330496 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f45f733-1397-4de5-81fd-b035baef4fb4-istio-token\" (UniqueName: \"kubernetes.io/projected/9f45f733-1397-4de5-81fd-b035baef4fb4-istio-token\") pod \"9f45f733-1397-4de5-81fd-b035baef4fb4\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " Oct 30 17:39:00.331257 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.331229 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-credential-socket\") pod \"9f45f733-1397-4de5-81fd-b035baef4fb4\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " Oct 30 17:39:00.331945 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.331928 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-istio-envoy\") pod \"9f45f733-1397-4de5-81fd-b035baef4fb4\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " Oct 30 17:39:00.332555 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.332535 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f45f733-1397-4de5-81fd-b035baef4fb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f45f733-1397-4de5-81fd-b035baef4fb4-istio-podinfo\") pod \"9f45f733-1397-4de5-81fd-b035baef4fb4\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " Oct 30 17:39:00.333037 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.333019 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-workload-socket\") pod \"9f45f733-1397-4de5-81fd-b035baef4fb4\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " Oct 30 17:39:00.333580 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.331841 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-credential-socket" pod "9f45f733-1397-4de5-81fd-b035baef4fb4" (UID: "9f45f733-1397-4de5-81fd-b035baef4fb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:00.333716 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.333399 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-workload-socket" pod "9f45f733-1397-4de5-81fd-b035baef4fb4" (UID: "9f45f733-1397-4de5-81fd-b035baef4fb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:00.333872 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.333843 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-workload-certs\") pod \"9f45f733-1397-4de5-81fd-b035baef4fb4\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " Oct 30 17:39:00.334267 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.334251 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f45f733-1397-4de5-81fd-b035baef4fb4-kube-api-access-jvctd\" (UniqueName: \"kubernetes.io/projected/9f45f733-1397-4de5-81fd-b035baef4fb4-kube-api-access-jvctd\") pod \"9f45f733-1397-4de5-81fd-b035baef4fb4\" (UID: \"9f45f733-1397-4de5-81fd-b035baef4fb4\") " Oct 30 17:39:00.334790 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.334181 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-workload-certs" pod "9f45f733-1397-4de5-81fd-b035baef4fb4" (UID: "9f45f733-1397-4de5-81fd-b035baef4fb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:00.336003 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.335983 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:39:00.336133 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.336120 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:39:00.336233 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.336222 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f45f733-1397-4de5-81fd-b035baef4fb4-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:39:00.336324 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.336315 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:39:00.336443 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.336432 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:39:00.342853 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.342831 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz"] Oct 30 17:39:00.343808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.343789 4965 cpu_manager.go:414] "containerMap: removing container" podUID="9f45f733-1397-4de5-81fd-b035baef4fb4" containerName="istio-proxy" Oct 30 17:39:00.343936 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.343808 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f45f733-1397-4de5-81fd-b035baef4fb4" containerName="istio-proxy" Oct 30 17:39:00.344042 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.344027 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f45f733-1397-4de5-81fd-b035baef4fb4" containerName="istio-proxy" Oct 30 17:39:00.348539 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.348523 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:00.369277 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.369244 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-802bda-openshift-default-dockercfg-699cp\"" Oct 30 17:39:00.382436 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.382392 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz"] Oct 30 17:39:00.412582 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.412534 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f45f733-1397-4de5-81fd-b035baef4fb4-istio-token" pod "9f45f733-1397-4de5-81fd-b035baef4fb4" (UID: "9f45f733-1397-4de5-81fd-b035baef4fb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:00.414481 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.414454 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-istio-envoy" pod "9f45f733-1397-4de5-81fd-b035baef4fb4" (UID: "9f45f733-1397-4de5-81fd-b035baef4fb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:00.415275 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.415226 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f45f733-1397-4de5-81fd-b035baef4fb4-kube-api-access-jvctd" pod "9f45f733-1397-4de5-81fd-b035baef4fb4" (UID: "9f45f733-1397-4de5-81fd-b035baef4fb4"). InnerVolumeSpecName "kube-api-access-jvctd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:00.416454 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.416429 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f45f733-1397-4de5-81fd-b035baef4fb4-istio-podinfo" pod "9f45f733-1397-4de5-81fd-b035baef4fb4" (UID: "9f45f733-1397-4de5-81fd-b035baef4fb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:39:00.439419 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.439065 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:39:00.441264 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.440237 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-credential-socket\") pod \"e2e-api-a-802bda-openshift-default-664b986cf5-689jz\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:00.441264 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.440327 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-workload-certs\") pod \"e2e-api-a-802bda-openshift-default-664b986cf5-689jz\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:00.441264 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.440576 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7be7daf-3252-4355-99da-2cd6c664b3ae-istio-token\") pod \"e2e-api-a-802bda-openshift-default-664b986cf5-689jz\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:00.441264 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.440617 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-istio-data\") pod \"e2e-api-a-802bda-openshift-default-664b986cf5-689jz\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:00.441264 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.440677 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-istio-envoy\") pod \"e2e-api-a-802bda-openshift-default-664b986cf5-689jz\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:00.441264 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.440774 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7be7daf-3252-4355-99da-2cd6c664b3ae-istiod-ca-cert\") pod \"e2e-api-a-802bda-openshift-default-664b986cf5-689jz\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:00.441264 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.440919 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmffg\" (UniqueName: \"kubernetes.io/projected/c7be7daf-3252-4355-99da-2cd6c664b3ae-kube-api-access-cmffg\") pod \"e2e-api-a-802bda-openshift-default-664b986cf5-689jz\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:00.441264 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.440971 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7be7daf-3252-4355-99da-2cd6c664b3ae-istio-podinfo\") pod \"e2e-api-a-802bda-openshift-default-664b986cf5-689jz\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:00.441264 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.441117 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-workload-socket\") pod \"e2e-api-a-802bda-openshift-default-664b986cf5-689jz\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:00.441264 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.441203 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f45f733-1397-4de5-81fd-b035baef4fb4-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:39:00.441264 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.441219 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f45f733-1397-4de5-81fd-b035baef4fb4-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:39:00.441264 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.441228 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvctd\" (UniqueName: \"kubernetes.io/projected/9f45f733-1397-4de5-81fd-b035baef4fb4-kube-api-access-jvctd\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:39:00.441264 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.441240 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f45f733-1397-4de5-81fd-b035baef4fb4-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:39:00.447125 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.447104 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:39:00.676950 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.549527 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c868037-59d9-480a-87c1-f29251d35284-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c868037-59d9-480a-87c1-f29251d35284-istio-podinfo\") pod \"5c868037-59d9-480a-87c1-f29251d35284\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " Oct 30 17:39:00.676950 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.613008 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c868037-59d9-480a-87c1-f29251d35284-istio-podinfo" pod "5c868037-59d9-480a-87c1-f29251d35284" (UID: "5c868037-59d9-480a-87c1-f29251d35284"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:39:00.704361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.677631 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-istio-envoy\") pod \"f4953033-0273-4681-8d46-94128ab21e6c\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " Oct 30 17:39:00.704361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.677674 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-istio-envoy\") pod \"5c868037-59d9-480a-87c1-f29251d35284\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " Oct 30 17:39:00.704361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.677751 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c868037-59d9-480a-87c1-f29251d35284-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c868037-59d9-480a-87c1-f29251d35284-istiod-ca-cert\") pod \"5c868037-59d9-480a-87c1-f29251d35284\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " Oct 30 17:39:00.704361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.677843 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4953033-0273-4681-8d46-94128ab21e6c-istio-token\" (UniqueName: \"kubernetes.io/projected/f4953033-0273-4681-8d46-94128ab21e6c-istio-token\") pod \"f4953033-0273-4681-8d46-94128ab21e6c\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " Oct 30 17:39:00.704361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.677909 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-workload-socket\") pod \"5c868037-59d9-480a-87c1-f29251d35284\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " Oct 30 17:39:00.704361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.677936 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-workload-certs\") pod \"f4953033-0273-4681-8d46-94128ab21e6c\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " Oct 30 17:39:00.704361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.677986 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4953033-0273-4681-8d46-94128ab21e6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4953033-0273-4681-8d46-94128ab21e6c-istiod-ca-cert\") pod \"f4953033-0273-4681-8d46-94128ab21e6c\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " Oct 30 17:39:00.704361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.678015 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-istio-data\") pod \"f4953033-0273-4681-8d46-94128ab21e6c\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " Oct 30 17:39:00.704361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.678127 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-workload-certs\") pod \"5c868037-59d9-480a-87c1-f29251d35284\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " Oct 30 17:39:00.704361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.678161 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4953033-0273-4681-8d46-94128ab21e6c-kube-api-access-5jwq4\" (UniqueName: \"kubernetes.io/projected/f4953033-0273-4681-8d46-94128ab21e6c-kube-api-access-5jwq4\") pod \"f4953033-0273-4681-8d46-94128ab21e6c\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " Oct 30 17:39:00.704361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.678208 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-istio-data\") pod \"5c868037-59d9-480a-87c1-f29251d35284\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " Oct 30 17:39:00.704361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.678250 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c868037-59d9-480a-87c1-f29251d35284-kube-api-access-rtqr5\" (UniqueName: \"kubernetes.io/projected/5c868037-59d9-480a-87c1-f29251d35284-kube-api-access-rtqr5\") pod \"5c868037-59d9-480a-87c1-f29251d35284\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " Oct 30 17:39:00.704361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.678276 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-credential-socket\") pod \"f4953033-0273-4681-8d46-94128ab21e6c\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " Oct 30 17:39:00.704361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.678305 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-workload-socket\") pod \"f4953033-0273-4681-8d46-94128ab21e6c\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " Oct 30 17:39:00.704361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.678395 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4953033-0273-4681-8d46-94128ab21e6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4953033-0273-4681-8d46-94128ab21e6c-istio-podinfo\") pod \"f4953033-0273-4681-8d46-94128ab21e6c\" (UID: \"f4953033-0273-4681-8d46-94128ab21e6c\") " Oct 30 17:39:00.704361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.678535 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-credential-socket\") pod \"5c868037-59d9-480a-87c1-f29251d35284\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " Oct 30 17:39:00.704361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.678711 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c868037-59d9-480a-87c1-f29251d35284-istio-token\" (UniqueName: \"kubernetes.io/projected/5c868037-59d9-480a-87c1-f29251d35284-istio-token\") pod \"5c868037-59d9-480a-87c1-f29251d35284\" (UID: \"5c868037-59d9-480a-87c1-f29251d35284\") " Oct 30 17:39:00.704361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.679092 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmffg\" (UniqueName: \"kubernetes.io/projected/c7be7daf-3252-4355-99da-2cd6c664b3ae-kube-api-access-cmffg\") pod \"e2e-api-a-802bda-openshift-default-664b986cf5-689jz\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:00.704361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.679155 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7be7daf-3252-4355-99da-2cd6c664b3ae-istio-podinfo\") pod \"e2e-api-a-802bda-openshift-default-664b986cf5-689jz\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:00.704361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.679218 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-workload-socket\") pod \"e2e-api-a-802bda-openshift-default-664b986cf5-689jz\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:00.704361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.679354 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-credential-socket\") pod \"e2e-api-a-802bda-openshift-default-664b986cf5-689jz\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:00.704361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.679447 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-workload-certs\") pod \"e2e-api-a-802bda-openshift-default-664b986cf5-689jz\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:00.704361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.679540 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7be7daf-3252-4355-99da-2cd6c664b3ae-istio-token\") pod \"e2e-api-a-802bda-openshift-default-664b986cf5-689jz\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:00.704361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.679574 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-istio-data\") pod \"e2e-api-a-802bda-openshift-default-664b986cf5-689jz\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:00.704361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.679648 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-istio-envoy\") pod \"e2e-api-a-802bda-openshift-default-664b986cf5-689jz\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:00.704361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.679778 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7be7daf-3252-4355-99da-2cd6c664b3ae-istiod-ca-cert\") pod \"e2e-api-a-802bda-openshift-default-664b986cf5-689jz\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:00.704361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.680024 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c868037-59d9-480a-87c1-f29251d35284-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:39:00.704361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.680793 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7be7daf-3252-4355-99da-2cd6c664b3ae-istiod-ca-cert\") pod \"e2e-api-a-802bda-openshift-default-664b986cf5-689jz\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:00.734249 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.708886 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-workload-socket\") pod \"e2e-api-a-802bda-openshift-default-664b986cf5-689jz\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:00.734249 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.715001 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-workload-certs" pod "5c868037-59d9-480a-87c1-f29251d35284" (UID: "5c868037-59d9-480a-87c1-f29251d35284"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:00.734249 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.733802 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-workload-socket" pod "5c868037-59d9-480a-87c1-f29251d35284" (UID: "5c868037-59d9-480a-87c1-f29251d35284"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:00.734249 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.734186 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c868037-59d9-480a-87c1-f29251d35284-istiod-ca-cert" pod "5c868037-59d9-480a-87c1-f29251d35284" (UID: "5c868037-59d9-480a-87c1-f29251d35284"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:39:00.754960 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.736041 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-workload-socket" pod "f4953033-0273-4681-8d46-94128ab21e6c" (UID: "f4953033-0273-4681-8d46-94128ab21e6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:00.754960 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.741202 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4953033-0273-4681-8d46-94128ab21e6c-istiod-ca-cert" pod "f4953033-0273-4681-8d46-94128ab21e6c" (UID: "f4953033-0273-4681-8d46-94128ab21e6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:39:00.754960 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.754089 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-workload-certs" pod "f4953033-0273-4681-8d46-94128ab21e6c" (UID: "f4953033-0273-4681-8d46-94128ab21e6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:00.754960 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.754263 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-istio-data" pod "f4953033-0273-4681-8d46-94128ab21e6c" (UID: "f4953033-0273-4681-8d46-94128ab21e6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:00.754960 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.754424 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-credential-socket" pod "5c868037-59d9-480a-87c1-f29251d35284" (UID: "5c868037-59d9-480a-87c1-f29251d35284"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:00.755480 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.755458 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-istio-data" pod "5c868037-59d9-480a-87c1-f29251d35284" (UID: "5c868037-59d9-480a-87c1-f29251d35284"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:00.802973 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.773214 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c868037-59d9-480a-87c1-f29251d35284-kube-api-access-rtqr5" pod "5c868037-59d9-480a-87c1-f29251d35284" (UID: "5c868037-59d9-480a-87c1-f29251d35284"). InnerVolumeSpecName "kube-api-access-rtqr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:00.802973 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.798687 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7be7daf-3252-4355-99da-2cd6c664b3ae-istio-podinfo\") pod \"e2e-api-a-802bda-openshift-default-664b986cf5-689jz\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:00.802973 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.801805 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-credential-socket\") pod \"e2e-api-a-802bda-openshift-default-664b986cf5-689jz\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:00.802973 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.802497 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-credential-socket" pod "f4953033-0273-4681-8d46-94128ab21e6c" (UID: "f4953033-0273-4681-8d46-94128ab21e6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:00.802973 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.802768 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-istio-data\") pod \"e2e-api-a-802bda-openshift-default-664b986cf5-689jz\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:00.806899 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.803815 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-workload-certs\") pod \"e2e-api-a-802bda-openshift-default-664b986cf5-689jz\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:00.807407 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.807114 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c868037-59d9-480a-87c1-f29251d35284-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:39:00.807407 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.807160 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:39:00.807407 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.807176 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:39:00.807407 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.807193 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4953033-0273-4681-8d46-94128ab21e6c-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:39:00.807407 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.807210 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:39:00.807407 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.807225 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:39:00.807407 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.807239 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:39:00.807407 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.807253 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtqr5\" (UniqueName: \"kubernetes.io/projected/5c868037-59d9-480a-87c1-f29251d35284-kube-api-access-rtqr5\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:39:00.807407 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.807266 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:39:00.807407 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.807286 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:39:00.807407 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.807304 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:39:00.820592 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.820534 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4953033-0273-4681-8d46-94128ab21e6c-istio-token" pod "f4953033-0273-4681-8d46-94128ab21e6c" (UID: "f4953033-0273-4681-8d46-94128ab21e6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:00.822728 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.822682 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmffg\" (UniqueName: \"kubernetes.io/projected/c7be7daf-3252-4355-99da-2cd6c664b3ae-kube-api-access-cmffg\") pod \"e2e-api-a-802bda-openshift-default-664b986cf5-689jz\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:00.823115 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.823092 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4953033-0273-4681-8d46-94128ab21e6c-kube-api-access-5jwq4" pod "f4953033-0273-4681-8d46-94128ab21e6c" (UID: "f4953033-0273-4681-8d46-94128ab21e6c"). InnerVolumeSpecName "kube-api-access-5jwq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:00.823397 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.823368 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-istio-envoy" pod "f4953033-0273-4681-8d46-94128ab21e6c" (UID: "f4953033-0273-4681-8d46-94128ab21e6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:00.823608 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.823589 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4953033-0273-4681-8d46-94128ab21e6c-istio-podinfo" pod "f4953033-0273-4681-8d46-94128ab21e6c" (UID: "f4953033-0273-4681-8d46-94128ab21e6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:39:00.823818 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.823793 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c868037-59d9-480a-87c1-f29251d35284-istio-token" pod "5c868037-59d9-480a-87c1-f29251d35284" (UID: "5c868037-59d9-480a-87c1-f29251d35284"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:00.826741 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.826677 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-istio-envoy\") pod \"e2e-api-a-802bda-openshift-default-664b986cf5-689jz\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:00.830976 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.830905 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7be7daf-3252-4355-99da-2cd6c664b3ae-istio-token\") pod \"e2e-api-a-802bda-openshift-default-664b986cf5-689jz\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:00.878101 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.878058 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-istio-envoy" pod "5c868037-59d9-480a-87c1-f29251d35284" (UID: "5c868037-59d9-480a-87c1-f29251d35284"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:00.958998 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.957085 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4953033-0273-4681-8d46-94128ab21e6c-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:39:00.958998 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.957123 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c868037-59d9-480a-87c1-f29251d35284-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:39:00.958998 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.957137 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4953033-0273-4681-8d46-94128ab21e6c-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:39:00.958998 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.957149 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c868037-59d9-480a-87c1-f29251d35284-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:39:00.958998 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.957162 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4953033-0273-4681-8d46-94128ab21e6c-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:39:00.958998 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:00.957175 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jwq4\" (UniqueName: \"kubernetes.io/projected/f4953033-0273-4681-8d46-94128ab21e6c-kube-api-access-5jwq4\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:39:01.092950 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:01.092913 4965 generic.go:356] "Generic (PLEG): container finished" podID="5c868037-59d9-480a-87c1-f29251d35284" containerID="499923eb9585afcca4e0c93b287ca973a017b039f0c24a4f56aee25e2ce88d67" exitCode=0 Oct 30 17:39:01.093401 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:01.093372 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" event={"ID":"5c868037-59d9-480a-87c1-f29251d35284","Type":"ContainerDied","Data":"499923eb9585afcca4e0c93b287ca973a017b039f0c24a4f56aee25e2ce88d67"} Oct 30 17:39:01.093595 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:01.093572 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" event={"ID":"5c868037-59d9-480a-87c1-f29251d35284","Type":"ContainerDied","Data":"26c6ef068c69a0498b1f1366197d28e69a1d86b009f63cd09c8e69895cf05ce1"} Oct 30 17:39:01.093734 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:01.093719 4965 scope.go:122] "RemoveContainer" containerID="499923eb9585afcca4e0c93b287ca973a017b039f0c24a4f56aee25e2ce88d67" Oct 30 17:39:01.094024 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:01.094004 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp" Oct 30 17:39:01.143857 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:01.143553 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:01.166895 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:01.165890 4965 generic.go:356] "Generic (PLEG): container finished" podID="f4953033-0273-4681-8d46-94128ab21e6c" containerID="aa9c503d179ee92a95cc9bc385e86b667d072ce5f0deb021da3cbf0531623bc2" exitCode=0 Oct 30 17:39:01.167983 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:01.167196 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" event={"ID":"f4953033-0273-4681-8d46-94128ab21e6c","Type":"ContainerDied","Data":"aa9c503d179ee92a95cc9bc385e86b667d072ce5f0deb021da3cbf0531623bc2"} Oct 30 17:39:01.167983 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:01.167277 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" event={"ID":"f4953033-0273-4681-8d46-94128ab21e6c","Type":"ContainerDied","Data":"13c888bef9e5f669f9fdcd727b9a06861ef0930a384c5c6679b76e3de29fc313"} Oct 30 17:39:01.167983 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:01.167359 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4" Oct 30 17:39:01.176173 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:01.175492 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" event={"ID":"9f45f733-1397-4de5-81fd-b035baef4fb4","Type":"ContainerDied","Data":"aa3e5931457e4f98ebb87624d65b55740042f62c258de07e5eaf4a60499ea0db"} Oct 30 17:39:01.176173 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:01.175603 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq" Oct 30 17:39:01.180637 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:01.179707 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" event={"ID":"30e31aef-c00b-4418-b005-50b3689d2912","Type":"ContainerStarted","Data":"73f7e236f32f6ded0e7e16ad8dffebf7b745fef40661d5a5517477120842b5f0"} Oct 30 17:39:01.203955 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:01.203929 4965 scope.go:122] "RemoveContainer" containerID="499923eb9585afcca4e0c93b287ca973a017b039f0c24a4f56aee25e2ce88d67" Oct 30 17:39:01.204737 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:39:01.204590 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"499923eb9585afcca4e0c93b287ca973a017b039f0c24a4f56aee25e2ce88d67\": container with ID starting with 499923eb9585afcca4e0c93b287ca973a017b039f0c24a4f56aee25e2ce88d67 not found: ID does not exist" containerID="499923eb9585afcca4e0c93b287ca973a017b039f0c24a4f56aee25e2ce88d67" Oct 30 17:39:01.204737 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:01.204628 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"499923eb9585afcca4e0c93b287ca973a017b039f0c24a4f56aee25e2ce88d67"} err="failed to get container status \"499923eb9585afcca4e0c93b287ca973a017b039f0c24a4f56aee25e2ce88d67\": rpc error: code = NotFound desc = could not find container \"499923eb9585afcca4e0c93b287ca973a017b039f0c24a4f56aee25e2ce88d67\": container with ID starting with 499923eb9585afcca4e0c93b287ca973a017b039f0c24a4f56aee25e2ce88d67 not found: ID does not exist" Oct 30 17:39:01.204737 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:01.204654 4965 scope.go:122] "RemoveContainer" containerID="aa9c503d179ee92a95cc9bc385e86b667d072ce5f0deb021da3cbf0531623bc2" Oct 30 17:39:01.240640 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:01.239059 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:39:01.249205 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:01.249140 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:39:01.259743 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:01.259687 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" podStartSLOduration=3.259669019 podStartE2EDuration="3.259669019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:39:01.255973844 +0000 UTC m=+1599.308073804" watchObservedRunningTime="2026-10-30 17:39:01.259669019 +0000 UTC m=+1599.311768952" Oct 30 17:39:01.281702 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:01.280579 4965 scope.go:122] "RemoveContainer" containerID="aa9c503d179ee92a95cc9bc385e86b667d072ce5f0deb021da3cbf0531623bc2" Oct 30 17:39:01.281702 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:39:01.281308 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa9c503d179ee92a95cc9bc385e86b667d072ce5f0deb021da3cbf0531623bc2\": container with ID starting with aa9c503d179ee92a95cc9bc385e86b667d072ce5f0deb021da3cbf0531623bc2 not found: ID does not exist" containerID="aa9c503d179ee92a95cc9bc385e86b667d072ce5f0deb021da3cbf0531623bc2" Oct 30 17:39:01.281702 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:01.281381 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa9c503d179ee92a95cc9bc385e86b667d072ce5f0deb021da3cbf0531623bc2"} err="failed to get container status \"aa9c503d179ee92a95cc9bc385e86b667d072ce5f0deb021da3cbf0531623bc2\": rpc error: code = NotFound desc = could not find container \"aa9c503d179ee92a95cc9bc385e86b667d072ce5f0deb021da3cbf0531623bc2\": container with ID starting with aa9c503d179ee92a95cc9bc385e86b667d072ce5f0deb021da3cbf0531623bc2 not found: ID does not exist" Oct 30 17:39:01.281702 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:01.281406 4965 scope.go:122] "RemoveContainer" containerID="c55a6ba88b03e73fcc59835e1762a6237e0371c5e89c93dbe498707dda12691f" Oct 30 17:39:01.356083 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:01.356054 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq"] Oct 30 17:39:01.365056 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:01.365024 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-rlnxq"] Oct 30 17:39:01.401106 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:01.401070 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp"] Oct 30 17:39:01.410888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:01.410837 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-f4qzp"] Oct 30 17:39:01.671369 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:01.671337 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4"] Oct 30 17:39:01.901390 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:01.872842 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-ff2p4"] Oct 30 17:39:01.901390 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:01.881470 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:39:01.901390 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:01.881629 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:39:01.901390 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:01.881727 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:39:01.901390 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:01.898224 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz"] Oct 30 17:39:01.992221 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:39:01.992164 4965 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f45f733_1397_4de5_81fd_b035baef4fb4.slice/crio-aa3e5931457e4f98ebb87624d65b55740042f62c258de07e5eaf4a60499ea0db\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c868037_59d9_480a_87c1_f29251d35284.slice/crio-26c6ef068c69a0498b1f1366197d28e69a1d86b009f63cd09c8e69895cf05ce1\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4953033_0273_4681_8d46_94128ab21e6c.slice/crio-13c888bef9e5f669f9fdcd727b9a06861ef0930a384c5c6679b76e3de29fc313\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4953033_0273_4681_8d46_94128ab21e6c.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f45f733_1397_4de5_81fd_b035baef4fb4.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c868037_59d9_480a_87c1_f29251d35284.slice\": RecentStats: unable to find data in memory cache]" Oct 30 17:39:02.123231 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:02.123197 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c868037-59d9-480a-87c1-f29251d35284" path="/var/lib/kubelet/pods/5c868037-59d9-480a-87c1-f29251d35284/volumes" Oct 30 17:39:02.124323 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:02.124299 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f45f733-1397-4de5-81fd-b035baef4fb4" path="/var/lib/kubelet/pods/9f45f733-1397-4de5-81fd-b035baef4fb4/volumes" Oct 30 17:39:02.125210 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:02.125190 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4953033-0273-4681-8d46-94128ab21e6c" path="/var/lib/kubelet/pods/f4953033-0273-4681-8d46-94128ab21e6c/volumes" Oct 30 17:39:02.233221 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:02.231961 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" event={"ID":"c7be7daf-3252-4355-99da-2cd6c664b3ae","Type":"ContainerStarted","Data":"408a5544a685adbd16ef188d221b11569357182c675be37b22f76d5be83094ca"} Oct 30 17:39:02.233221 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:02.232028 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:39:02.233221 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:02.232061 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" event={"ID":"c7be7daf-3252-4355-99da-2cd6c664b3ae","Type":"ContainerStarted","Data":"b3e76a2eef4a6ee1737284f2b06ac58dd5d95928d377f7d42af43825b8782002"} Oct 30 17:39:02.244325 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:02.239394 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:39:02.306895 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:02.301235 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" podStartSLOduration=2.301215311 podStartE2EDuration="2.301215311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:39:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:39:02.298066891 +0000 UTC m=+1600.350166824" watchObservedRunningTime="2026-10-30 17:39:02.301215311 +0000 UTC m=+1600.353315243" Oct 30 17:39:03.156233 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:03.154201 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:03.187592 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:03.187145 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:03.280894 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:03.277679 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:03.286635 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:03.284077 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:39:03.694886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:03.693966 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:04.024669 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:04.015839 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:04.062886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:04.062689 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:04.346918 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:04.341615 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" podUID="7715de9f-65c1-44fb-b83a-692ec07b30a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": context deadline exceeded" Oct 30 17:39:04.493880 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:04.493804 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" podUID="7715de9f-65c1-44fb-b83a-692ec07b30a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/health\": context deadline exceeded" Oct 30 17:39:04.494558 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:04.494514 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:04.543088 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:04.543028 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" podUID="d174a74f-7205-4020-94b0-e30652c9b721" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:04.613946 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:04.612158 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 17:39:04.710245 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:04.710182 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:04.711012 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:04.710987 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:39:04.735978 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:04.735481 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:39:05.926112 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:05.926075 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c9bdd9-5748f769dd-wl8td"] Oct 30 17:39:05.926904 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:05.926881 4965 cpu_manager.go:414] "containerMap: removing container" podUID="f4953033-0273-4681-8d46-94128ab21e6c" containerName="istio-proxy" Oct 30 17:39:05.926904 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:05.926898 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4953033-0273-4681-8d46-94128ab21e6c" containerName="istio-proxy" Oct 30 17:39:05.926988 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:05.926924 4965 cpu_manager.go:414] "containerMap: removing container" podUID="5c868037-59d9-480a-87c1-f29251d35284" containerName="istio-proxy" Oct 30 17:39:05.926988 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:05.926929 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c868037-59d9-480a-87c1-f29251d35284" containerName="istio-proxy" Oct 30 17:39:05.927066 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:05.927045 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4953033-0273-4681-8d46-94128ab21e6c" containerName="istio-proxy" Oct 30 17:39:05.927066 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:05.927056 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c868037-59d9-480a-87c1-f29251d35284" containerName="istio-proxy" Oct 30 17:39:05.932325 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:05.932304 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c9bdd9-5748f769dd-wl8td" Oct 30 17:39:05.949005 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:05.948982 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c9bdd9-5748f769dd-wl8td"] Oct 30 17:39:05.956652 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:05.956631 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c9bdd9\"" Oct 30 17:39:05.956798 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:05.956775 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c9bdd9-dockercfg-qmw6f\"" Oct 30 17:39:06.001317 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:39:06.001284 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:39:06.002748 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:06.002721 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c9bdd9-56b7d67c7f-r2hzh"] Oct 30 17:39:06.008729 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:06.008705 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9bdd9-56b7d67c7f-r2hzh" Oct 30 17:39:06.022965 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:06.022936 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c9bdd9-56b7d67c7f-r2hzh"] Oct 30 17:39:06.265944 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:06.257679 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jmvw\" (UniqueName: \"kubernetes.io/projected/174ed90c-b452-485c-9dfb-af6a167089b0-kube-api-access-8jmvw\") pod \"payload-processing-e2e-api-a-c9bdd9-5748f769dd-wl8td\" (UID: \"174ed90c-b452-485c-9dfb-af6a167089b0\") " pod="openshift-ingress/payload-processing-e2e-api-a-c9bdd9-5748f769dd-wl8td" Oct 30 17:39:06.265944 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:06.257822 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlbdl\" (UniqueName: \"kubernetes.io/projected/9960af67-a570-4962-a0ac-770879a98949-kube-api-access-dlbdl\") pod \"payload-pre-processing-e2e-api-a-c9bdd9-56b7d67c7f-r2hzh\" (UID: \"9960af67-a570-4962-a0ac-770879a98949\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9bdd9-56b7d67c7f-r2hzh" Oct 30 17:39:06.265944 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:06.257981 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9960af67-a570-4962-a0ac-770879a98949-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c9bdd9-56b7d67c7f-r2hzh\" (UID: \"9960af67-a570-4962-a0ac-770879a98949\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9bdd9-56b7d67c7f-r2hzh" Oct 30 17:39:06.265944 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:06.258022 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/174ed90c-b452-485c-9dfb-af6a167089b0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c9bdd9-5748f769dd-wl8td\" (UID: \"174ed90c-b452-485c-9dfb-af6a167089b0\") " pod="openshift-ingress/payload-processing-e2e-api-a-c9bdd9-5748f769dd-wl8td" Oct 30 17:39:06.374452 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:06.365389 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlbdl\" (UniqueName: \"kubernetes.io/projected/9960af67-a570-4962-a0ac-770879a98949-kube-api-access-dlbdl\") pod \"payload-pre-processing-e2e-api-a-c9bdd9-56b7d67c7f-r2hzh\" (UID: \"9960af67-a570-4962-a0ac-770879a98949\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9bdd9-56b7d67c7f-r2hzh" Oct 30 17:39:06.374452 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:06.365594 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9960af67-a570-4962-a0ac-770879a98949-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c9bdd9-56b7d67c7f-r2hzh\" (UID: \"9960af67-a570-4962-a0ac-770879a98949\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9bdd9-56b7d67c7f-r2hzh" Oct 30 17:39:06.374452 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:06.365639 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/174ed90c-b452-485c-9dfb-af6a167089b0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c9bdd9-5748f769dd-wl8td\" (UID: \"174ed90c-b452-485c-9dfb-af6a167089b0\") " pod="openshift-ingress/payload-processing-e2e-api-a-c9bdd9-5748f769dd-wl8td" Oct 30 17:39:06.374452 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:06.365757 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jmvw\" (UniqueName: \"kubernetes.io/projected/174ed90c-b452-485c-9dfb-af6a167089b0-kube-api-access-8jmvw\") pod \"payload-processing-e2e-api-a-c9bdd9-5748f769dd-wl8td\" (UID: \"174ed90c-b452-485c-9dfb-af6a167089b0\") " pod="openshift-ingress/payload-processing-e2e-api-a-c9bdd9-5748f769dd-wl8td" Oct 30 17:39:06.376783 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:06.376748 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9960af67-a570-4962-a0ac-770879a98949-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c9bdd9-56b7d67c7f-r2hzh\" (UID: \"9960af67-a570-4962-a0ac-770879a98949\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9bdd9-56b7d67c7f-r2hzh" Oct 30 17:39:06.377634 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:06.377597 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/174ed90c-b452-485c-9dfb-af6a167089b0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c9bdd9-5748f769dd-wl8td\" (UID: \"174ed90c-b452-485c-9dfb-af6a167089b0\") " pod="openshift-ingress/payload-processing-e2e-api-a-c9bdd9-5748f769dd-wl8td" Oct 30 17:39:06.458358 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:06.458144 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlbdl\" (UniqueName: \"kubernetes.io/projected/9960af67-a570-4962-a0ac-770879a98949-kube-api-access-dlbdl\") pod \"payload-pre-processing-e2e-api-a-c9bdd9-56b7d67c7f-r2hzh\" (UID: \"9960af67-a570-4962-a0ac-770879a98949\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9bdd9-56b7d67c7f-r2hzh" Oct 30 17:39:06.460034 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:06.459977 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jmvw\" (UniqueName: \"kubernetes.io/projected/174ed90c-b452-485c-9dfb-af6a167089b0-kube-api-access-8jmvw\") pod \"payload-processing-e2e-api-a-c9bdd9-5748f769dd-wl8td\" (UID: \"174ed90c-b452-485c-9dfb-af6a167089b0\") " pod="openshift-ingress/payload-processing-e2e-api-a-c9bdd9-5748f769dd-wl8td" Oct 30 17:39:06.486823 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:06.486773 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7bb2e69d-54765d4d8c-kvxq7"] Oct 30 17:39:06.548735 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:06.548654 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c9bdd9-5748f769dd-wl8td" Oct 30 17:39:06.665614 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:06.659981 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9bdd9-56b7d67c7f-r2hzh" Oct 30 17:39:07.345365 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:07.345328 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c9bdd9-5748f769dd-wl8td"] Oct 30 17:39:07.365943 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:39:07.365895 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod174ed90c_b452_485c_9dfb_af6a167089b0.slice/crio-a424fce8dc925678f8891ad7cfc58e26fa2f17d606dda0d23c7ddba274c96f2b WatchSource:0}: Error finding container a424fce8dc925678f8891ad7cfc58e26fa2f17d606dda0d23c7ddba274c96f2b: Status 404 returned error can't find the container with id a424fce8dc925678f8891ad7cfc58e26fa2f17d606dda0d23c7ddba274c96f2b Oct 30 17:39:07.395149 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:07.395093 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c9bdd9-56b7d67c7f-r2hzh"] Oct 30 17:39:07.411992 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:07.411932 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c9bdd9-5748f769dd-wl8td" event={"ID":"174ed90c-b452-485c-9dfb-af6a167089b0","Type":"ContainerStarted","Data":"a424fce8dc925678f8891ad7cfc58e26fa2f17d606dda0d23c7ddba274c96f2b"} Oct 30 17:39:08.422926 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:08.422851 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c9bdd9-5748f769dd-wl8td" event={"ID":"174ed90c-b452-485c-9dfb-af6a167089b0","Type":"ContainerStarted","Data":"634e281bf586356d2ee7b469b770959c2542d90ffaf0f61fa49b6f894c2c75d9"} Oct 30 17:39:08.423424 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:08.422964 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c9bdd9-5748f769dd-wl8td" Oct 30 17:39:08.424800 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:08.424764 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9bdd9-56b7d67c7f-r2hzh" event={"ID":"9960af67-a570-4962-a0ac-770879a98949","Type":"ContainerStarted","Data":"2bcfb1fb6157b56a7dbda3713b6930f8dc925de8ea50639d593f78e9c13150cb"} Oct 30 17:39:08.424940 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:08.424808 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9bdd9-56b7d67c7f-r2hzh" event={"ID":"9960af67-a570-4962-a0ac-770879a98949","Type":"ContainerStarted","Data":"dc426b12e8990109e9a810efc10d1f6e002c28afd940fe1901d2b38b98804619"} Oct 30 17:39:08.424940 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:08.424848 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9bdd9-56b7d67c7f-r2hzh" Oct 30 17:39:08.464371 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:08.464324 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c9bdd9-5748f769dd-wl8td" podStartSLOduration=3.464310917 podStartE2EDuration="3.464310917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:39:08.463854819 +0000 UTC m=+1606.515954750" watchObservedRunningTime="2026-10-30 17:39:08.464310917 +0000 UTC m=+1606.516410848" Oct 30 17:39:08.511421 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:08.511385 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-802bda-6cbf7f6d44-fnfr8"] Oct 30 17:39:08.512070 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:08.512031 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9bdd9-56b7d67c7f-r2hzh" podStartSLOduration=3.512018017 podStartE2EDuration="3.512018017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:39:08.509553177 +0000 UTC m=+1606.561653119" watchObservedRunningTime="2026-10-30 17:39:08.512018017 +0000 UTC m=+1606.564117947" Oct 30 17:39:08.517234 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:08.517211 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-802bda-6cbf7f6d44-fnfr8" Oct 30 17:39:08.534317 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:08.534296 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-802bda\"" Oct 30 17:39:08.535229 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:08.535211 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-802bda-dockercfg-lch49\"" Oct 30 17:39:08.564110 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:08.564070 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-802bda-6cbf7f6d44-fnfr8"] Oct 30 17:39:08.586766 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:08.586732 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82j7w\" (UniqueName: \"kubernetes.io/projected/704f5086-b986-45c0-b515-a04fd6e0a0e0-kube-api-access-82j7w\") pod \"payload-processing-e2e-api-a-802bda-6cbf7f6d44-fnfr8\" (UID: \"704f5086-b986-45c0-b515-a04fd6e0a0e0\") " pod="openshift-ingress/payload-processing-e2e-api-a-802bda-6cbf7f6d44-fnfr8" Oct 30 17:39:08.587159 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:08.587133 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/704f5086-b986-45c0-b515-a04fd6e0a0e0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-802bda-6cbf7f6d44-fnfr8\" (UID: \"704f5086-b986-45c0-b515-a04fd6e0a0e0\") " pod="openshift-ingress/payload-processing-e2e-api-a-802bda-6cbf7f6d44-fnfr8" Oct 30 17:39:08.612255 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:08.612223 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-802bda-55497ddf87-452b4"] Oct 30 17:39:08.617158 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:08.617138 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-802bda-55497ddf87-452b4" Oct 30 17:39:08.747319 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:08.706403 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/704f5086-b986-45c0-b515-a04fd6e0a0e0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-802bda-6cbf7f6d44-fnfr8\" (UID: \"704f5086-b986-45c0-b515-a04fd6e0a0e0\") " pod="openshift-ingress/payload-processing-e2e-api-a-802bda-6cbf7f6d44-fnfr8" Oct 30 17:39:08.759885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:08.759045 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zt7s\" (UniqueName: \"kubernetes.io/projected/e9ebc66a-28f5-4a2e-ae0e-28933f360997-kube-api-access-5zt7s\") pod \"payload-pre-processing-e2e-api-a-802bda-55497ddf87-452b4\" (UID: \"e9ebc66a-28f5-4a2e-ae0e-28933f360997\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-802bda-55497ddf87-452b4" Oct 30 17:39:08.759885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:08.759171 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82j7w\" (UniqueName: \"kubernetes.io/projected/704f5086-b986-45c0-b515-a04fd6e0a0e0-kube-api-access-82j7w\") pod \"payload-processing-e2e-api-a-802bda-6cbf7f6d44-fnfr8\" (UID: \"704f5086-b986-45c0-b515-a04fd6e0a0e0\") " pod="openshift-ingress/payload-processing-e2e-api-a-802bda-6cbf7f6d44-fnfr8" Oct 30 17:39:08.759885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:08.759459 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ebc66a-28f5-4a2e-ae0e-28933f360997-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-802bda-55497ddf87-452b4\" (UID: \"e9ebc66a-28f5-4a2e-ae0e-28933f360997\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-802bda-55497ddf87-452b4" Oct 30 17:39:08.759885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:08.718014 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/704f5086-b986-45c0-b515-a04fd6e0a0e0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-802bda-6cbf7f6d44-fnfr8\" (UID: \"704f5086-b986-45c0-b515-a04fd6e0a0e0\") " pod="openshift-ingress/payload-processing-e2e-api-a-802bda-6cbf7f6d44-fnfr8" Oct 30 17:39:08.884262 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:08.842633 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82j7w\" (UniqueName: \"kubernetes.io/projected/704f5086-b986-45c0-b515-a04fd6e0a0e0-kube-api-access-82j7w\") pod \"payload-processing-e2e-api-a-802bda-6cbf7f6d44-fnfr8\" (UID: \"704f5086-b986-45c0-b515-a04fd6e0a0e0\") " pod="openshift-ingress/payload-processing-e2e-api-a-802bda-6cbf7f6d44-fnfr8" Oct 30 17:39:08.884262 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:08.842976 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-802bda-6cbf7f6d44-fnfr8" Oct 30 17:39:08.884262 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:08.882988 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zt7s\" (UniqueName: \"kubernetes.io/projected/e9ebc66a-28f5-4a2e-ae0e-28933f360997-kube-api-access-5zt7s\") pod \"payload-pre-processing-e2e-api-a-802bda-55497ddf87-452b4\" (UID: \"e9ebc66a-28f5-4a2e-ae0e-28933f360997\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-802bda-55497ddf87-452b4" Oct 30 17:39:08.884262 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:08.883238 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ebc66a-28f5-4a2e-ae0e-28933f360997-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-802bda-55497ddf87-452b4\" (UID: \"e9ebc66a-28f5-4a2e-ae0e-28933f360997\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-802bda-55497ddf87-452b4" Oct 30 17:39:08.884619 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:08.884292 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ebc66a-28f5-4a2e-ae0e-28933f360997-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-802bda-55497ddf87-452b4\" (UID: \"e9ebc66a-28f5-4a2e-ae0e-28933f360997\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-802bda-55497ddf87-452b4" Oct 30 17:39:09.015904 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:09.012100 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-802bda-55497ddf87-452b4"] Oct 30 17:39:09.018558 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:09.018527 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zt7s\" (UniqueName: \"kubernetes.io/projected/e9ebc66a-28f5-4a2e-ae0e-28933f360997-kube-api-access-5zt7s\") pod \"payload-pre-processing-e2e-api-a-802bda-55497ddf87-452b4\" (UID: \"e9ebc66a-28f5-4a2e-ae0e-28933f360997\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-802bda-55497ddf87-452b4" Oct 30 17:39:09.049729 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:09.049676 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-df8b20-69845c8495-f4mq5"] Oct 30 17:39:09.278023 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:09.275076 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-802bda-55497ddf87-452b4" Oct 30 17:39:10.480817 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:39:10.480773 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704f5086_b986_45c0_b515_a04fd6e0a0e0.slice/crio-f4265ebe201400dea4a3495a6b54c59c79a3ba4f0f72ff9d12bb9464dd48d5cd WatchSource:0}: Error finding container f4265ebe201400dea4a3495a6b54c59c79a3ba4f0f72ff9d12bb9464dd48d5cd: Status 404 returned error can't find the container with id f4265ebe201400dea4a3495a6b54c59c79a3ba4f0f72ff9d12bb9464dd48d5cd Oct 30 17:39:10.481198 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:10.480940 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-802bda-6cbf7f6d44-fnfr8"] Oct 30 17:39:10.510703 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:10.510676 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-802bda-6cbf7f6d44-fnfr8" event={"ID":"704f5086-b986-45c0-b515-a04fd6e0a0e0","Type":"ContainerStarted","Data":"f4265ebe201400dea4a3495a6b54c59c79a3ba4f0f72ff9d12bb9464dd48d5cd"} Oct 30 17:39:10.535022 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:39:10.534996 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ebc66a_28f5_4a2e_ae0e_28933f360997.slice/crio-396456f6197f6ce397c100f8fcdd82d5cf5a5d2616c300f584d263710e205d42 WatchSource:0}: Error finding container 396456f6197f6ce397c100f8fcdd82d5cf5a5d2616c300f584d263710e205d42: Status 404 returned error can't find the container with id 396456f6197f6ce397c100f8fcdd82d5cf5a5d2616c300f584d263710e205d42 Oct 30 17:39:10.535318 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:10.535296 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-802bda-55497ddf87-452b4"] Oct 30 17:39:11.487496 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:11.487438 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7bb2e69d-54765d4d8c-kvxq7" podUID="a48121a5-ca41-4b77-a554-a08b069c97ea" containerName="payload-pre-processing" containerID="cri-o://662777fe077a2253d601ada61bbfcd106caf68dda79d8b2888307560dfad715d" gracePeriod=25 Oct 30 17:39:11.521089 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:11.521061 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-802bda-55497ddf87-452b4" event={"ID":"e9ebc66a-28f5-4a2e-ae0e-28933f360997","Type":"ContainerStarted","Data":"0f8f583735f82b5e8b628fa912144fe3af2c08e87cec45925659d554d858b123"} Oct 30 17:39:11.521224 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:11.521156 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-802bda-55497ddf87-452b4" event={"ID":"e9ebc66a-28f5-4a2e-ae0e-28933f360997","Type":"ContainerStarted","Data":"396456f6197f6ce397c100f8fcdd82d5cf5a5d2616c300f584d263710e205d42"} Oct 30 17:39:11.521295 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:11.521271 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-802bda-55497ddf87-452b4" Oct 30 17:39:11.522782 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:11.522760 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-802bda-6cbf7f6d44-fnfr8" event={"ID":"704f5086-b986-45c0-b515-a04fd6e0a0e0","Type":"ContainerStarted","Data":"21e34ea2ae10582e477be0edef02dd1ad14bc90c3b1996f5f01181ea4355d4b3"} Oct 30 17:39:11.522928 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:11.522913 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-802bda-6cbf7f6d44-fnfr8" Oct 30 17:39:11.571719 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:11.571667 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-802bda-55497ddf87-452b4" podStartSLOduration=3.571655913 podStartE2EDuration="3.571655913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:39:11.569022834 +0000 UTC m=+1609.621122765" watchObservedRunningTime="2026-10-30 17:39:11.571655913 +0000 UTC m=+1609.623755844" Oct 30 17:39:11.620352 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:11.620290 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-802bda-6cbf7f6d44-fnfr8" podStartSLOduration=3.62027169 podStartE2EDuration="3.62027169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:39:11.616954437 +0000 UTC m=+1609.669054379" watchObservedRunningTime="2026-10-30 17:39:11.62027169 +0000 UTC m=+1609.672371697" Oct 30 17:39:11.878084 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:11.878057 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7bb2e69d-54765d4d8c-kvxq7" Oct 30 17:39:12.043123 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:12.043092 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a48121a5-ca41-4b77-a554-a08b069c97ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a48121a5-ca41-4b77-a554-a08b069c97ea-plugins-config-volume\") pod \"a48121a5-ca41-4b77-a554-a08b069c97ea\" (UID: \"a48121a5-ca41-4b77-a554-a08b069c97ea\") " Oct 30 17:39:12.043323 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:12.043139 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a48121a5-ca41-4b77-a554-a08b069c97ea-kube-api-access-c5dmx\" (UniqueName: \"kubernetes.io/projected/a48121a5-ca41-4b77-a554-a08b069c97ea-kube-api-access-c5dmx\") pod \"a48121a5-ca41-4b77-a554-a08b069c97ea\" (UID: \"a48121a5-ca41-4b77-a554-a08b069c97ea\") " Oct 30 17:39:12.043508 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:12.043484 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a48121a5-ca41-4b77-a554-a08b069c97ea-plugins-config-volume" pod "a48121a5-ca41-4b77-a554-a08b069c97ea" (UID: "a48121a5-ca41-4b77-a554-a08b069c97ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:39:12.043841 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:12.043822 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a48121a5-ca41-4b77-a554-a08b069c97ea-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:39:12.045373 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:12.045347 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a48121a5-ca41-4b77-a554-a08b069c97ea-kube-api-access-c5dmx" pod "a48121a5-ca41-4b77-a554-a08b069c97ea" (UID: "a48121a5-ca41-4b77-a554-a08b069c97ea"). InnerVolumeSpecName "kube-api-access-c5dmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:12.145962 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:12.145924 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5dmx\" (UniqueName: \"kubernetes.io/projected/a48121a5-ca41-4b77-a554-a08b069c97ea-kube-api-access-c5dmx\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:39:12.532411 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:12.532374 4965 generic.go:356] "Generic (PLEG): container finished" podID="a48121a5-ca41-4b77-a554-a08b069c97ea" containerID="662777fe077a2253d601ada61bbfcd106caf68dda79d8b2888307560dfad715d" exitCode=0 Oct 30 17:39:12.532887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:12.532419 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7bb2e69d-54765d4d8c-kvxq7" Oct 30 17:39:12.532887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:12.532462 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7bb2e69d-54765d4d8c-kvxq7" event={"ID":"a48121a5-ca41-4b77-a554-a08b069c97ea","Type":"ContainerDied","Data":"662777fe077a2253d601ada61bbfcd106caf68dda79d8b2888307560dfad715d"} Oct 30 17:39:12.532887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:12.532527 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7bb2e69d-54765d4d8c-kvxq7" event={"ID":"a48121a5-ca41-4b77-a554-a08b069c97ea","Type":"ContainerDied","Data":"34f31ece8f16dadc9380f8cf436959daf6b6d22465b7cc98e372cf71f65c1122"} Oct 30 17:39:12.532887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:12.532556 4965 scope.go:122] "RemoveContainer" containerID="662777fe077a2253d601ada61bbfcd106caf68dda79d8b2888307560dfad715d" Oct 30 17:39:12.543450 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:12.543428 4965 scope.go:122] "RemoveContainer" containerID="662777fe077a2253d601ada61bbfcd106caf68dda79d8b2888307560dfad715d" Oct 30 17:39:12.543780 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:39:12.543760 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"662777fe077a2253d601ada61bbfcd106caf68dda79d8b2888307560dfad715d\": container with ID starting with 662777fe077a2253d601ada61bbfcd106caf68dda79d8b2888307560dfad715d not found: ID does not exist" containerID="662777fe077a2253d601ada61bbfcd106caf68dda79d8b2888307560dfad715d" Oct 30 17:39:12.543881 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:12.543791 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"662777fe077a2253d601ada61bbfcd106caf68dda79d8b2888307560dfad715d"} err="failed to get container status \"662777fe077a2253d601ada61bbfcd106caf68dda79d8b2888307560dfad715d\": rpc error: code = NotFound desc = could not find container \"662777fe077a2253d601ada61bbfcd106caf68dda79d8b2888307560dfad715d\": container with ID starting with 662777fe077a2253d601ada61bbfcd106caf68dda79d8b2888307560dfad715d not found: ID does not exist" Oct 30 17:39:12.586895 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:12.586834 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7bb2e69d-54765d4d8c-kvxq7"] Oct 30 17:39:12.598581 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:12.598556 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7bb2e69d-54765d4d8c-kvxq7"] Oct 30 17:39:12.802611 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:12.802516 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-7bb2e69d-54765d4d8c-kvxq7" podUID="a48121a5-ca41-4b77-a554-a08b069c97ea" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.108:50052\" within 1s: context deadline exceeded" Oct 30 17:39:12.802767 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:39:12.802628 4965 logging.go:55] [core] [Channel #1883 SubChannel #1884]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.108:50052", ServerName: "10.132.0.108:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.108:50052: operation was canceled" Oct 30 17:39:14.056928 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:14.055981 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df8b20-69845c8495-f4mq5" podUID="631b7466-f017-4430-89ee-f62b93d000bb" containerName="payload-pre-processing" containerID="cri-o://3d7b15492caa77bf4f405797ca7acd32570d24e393ee3004f826f97e5080c10b" gracePeriod=25 Oct 30 17:39:14.102829 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:14.102802 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a48121a5-ca41-4b77-a554-a08b069c97ea" path="/var/lib/kubelet/pods/a48121a5-ca41-4b77-a554-a08b069c97ea/volumes" Oct 30 17:39:14.397702 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:14.397676 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df8b20-69845c8495-f4mq5" Oct 30 17:39:14.498843 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:14.498805 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/631b7466-f017-4430-89ee-f62b93d000bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/631b7466-f017-4430-89ee-f62b93d000bb-plugins-config-volume\") pod \"631b7466-f017-4430-89ee-f62b93d000bb\" (UID: \"631b7466-f017-4430-89ee-f62b93d000bb\") " Oct 30 17:39:14.499025 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:14.498986 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/631b7466-f017-4430-89ee-f62b93d000bb-kube-api-access-xjb6n\" (UniqueName: \"kubernetes.io/projected/631b7466-f017-4430-89ee-f62b93d000bb-kube-api-access-xjb6n\") pod \"631b7466-f017-4430-89ee-f62b93d000bb\" (UID: \"631b7466-f017-4430-89ee-f62b93d000bb\") " Oct 30 17:39:14.499227 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:14.499202 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/631b7466-f017-4430-89ee-f62b93d000bb-plugins-config-volume" pod "631b7466-f017-4430-89ee-f62b93d000bb" (UID: "631b7466-f017-4430-89ee-f62b93d000bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:39:14.499543 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:14.499527 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/631b7466-f017-4430-89ee-f62b93d000bb-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:39:14.501105 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:14.501082 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/631b7466-f017-4430-89ee-f62b93d000bb-kube-api-access-xjb6n" pod "631b7466-f017-4430-89ee-f62b93d000bb" (UID: "631b7466-f017-4430-89ee-f62b93d000bb"). InnerVolumeSpecName "kube-api-access-xjb6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:14.583655 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:14.583617 4965 generic.go:356] "Generic (PLEG): container finished" podID="631b7466-f017-4430-89ee-f62b93d000bb" containerID="3d7b15492caa77bf4f405797ca7acd32570d24e393ee3004f826f97e5080c10b" exitCode=0 Oct 30 17:39:14.583790 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:14.583684 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df8b20-69845c8495-f4mq5" Oct 30 17:39:14.583790 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:14.583703 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df8b20-69845c8495-f4mq5" event={"ID":"631b7466-f017-4430-89ee-f62b93d000bb","Type":"ContainerDied","Data":"3d7b15492caa77bf4f405797ca7acd32570d24e393ee3004f826f97e5080c10b"} Oct 30 17:39:14.583908 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:14.583794 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df8b20-69845c8495-f4mq5" event={"ID":"631b7466-f017-4430-89ee-f62b93d000bb","Type":"ContainerDied","Data":"ffe76e440482f074c1931c78320016d3b6ab1ee713c760687313979027476d0c"} Oct 30 17:39:14.583908 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:14.583822 4965 scope.go:122] "RemoveContainer" containerID="3d7b15492caa77bf4f405797ca7acd32570d24e393ee3004f826f97e5080c10b" Oct 30 17:39:14.594903 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:14.594876 4965 scope.go:122] "RemoveContainer" containerID="3d7b15492caa77bf4f405797ca7acd32570d24e393ee3004f826f97e5080c10b" Oct 30 17:39:14.595174 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:39:14.595154 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d7b15492caa77bf4f405797ca7acd32570d24e393ee3004f826f97e5080c10b\": container with ID starting with 3d7b15492caa77bf4f405797ca7acd32570d24e393ee3004f826f97e5080c10b not found: ID does not exist" containerID="3d7b15492caa77bf4f405797ca7acd32570d24e393ee3004f826f97e5080c10b" Oct 30 17:39:14.595240 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:14.595187 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d7b15492caa77bf4f405797ca7acd32570d24e393ee3004f826f97e5080c10b"} err="failed to get container status \"3d7b15492caa77bf4f405797ca7acd32570d24e393ee3004f826f97e5080c10b\": rpc error: code = NotFound desc = could not find container \"3d7b15492caa77bf4f405797ca7acd32570d24e393ee3004f826f97e5080c10b\": container with ID starting with 3d7b15492caa77bf4f405797ca7acd32570d24e393ee3004f826f97e5080c10b not found: ID does not exist" Oct 30 17:39:14.600686 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:14.600663 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjb6n\" (UniqueName: \"kubernetes.io/projected/631b7466-f017-4430-89ee-f62b93d000bb-kube-api-access-xjb6n\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:39:14.637819 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:14.637743 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-df8b20-69845c8495-f4mq5"] Oct 30 17:39:14.642330 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:14.642305 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-df8b20-69845c8495-f4mq5"] Oct 30 17:39:16.121185 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:16.121150 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="631b7466-f017-4430-89ee-f62b93d000bb" path="/var/lib/kubelet/pods/631b7466-f017-4430-89ee-f62b93d000bb/volumes" Oct 30 17:39:19.506075 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:19.506046 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c9bdd9-5748f769dd-wl8td" Oct 30 17:39:19.540709 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:19.540685 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9bdd9-56b7d67c7f-r2hzh" Oct 30 17:39:22.536374 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:22.536346 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-802bda-55497ddf87-452b4" Oct 30 17:39:22.536744 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:22.536418 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-802bda-6cbf7f6d44-fnfr8" Oct 30 17:39:36.457345 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:36.452984 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:36.655273 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:39:36.655232 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:39:36.751224 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:36.751129 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:36.852801 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:36.852762 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-xmr9m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:39:36.853134 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:36.853087 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" podUID="d054b35e-8973-4d7b-84e5-6a49ac14bfe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:36.983886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:36.978247 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:37.048180 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:37.048031 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:37.651052 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:37.650980 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:37.820886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:37.819937 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:37.820886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:37.820413 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:39:37.915944 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:37.915368 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 17:39:56.701888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:39:56.686998 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:40:02.314366 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:02.313550 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d1c973-5d4f7449db-8dh8t"] Oct 30 17:40:02.318921 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:02.318893 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d1c973-5d95487bd7-vmdzr"] Oct 30 17:40:06.049547 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:40:06.049503 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:40:07.314944 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:07.314885 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d1c973-5d4f7449db-8dh8t" podUID="32b71d3c-848d-453a-8a75-5b3e0281050d" containerName="payload-processing" containerID="cri-o://f67e2fc7dcd3992c193c8c5616a6f952fe9ef47a55e8a062080c2e3268f885a1" gracePeriod=25 Oct 30 17:40:07.319527 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:07.319461 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1c973-5d95487bd7-vmdzr" podUID="743b47bf-ff68-4a25-bb33-785d836c1cc9" containerName="payload-pre-processing" containerID="cri-o://82888a930a858928e9f0489be629e964206ce2fdb9d19bd258f4354f867f1f66" gracePeriod=25 Oct 30 17:40:07.810410 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:07.810377 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d1c973-5d4f7449db-8dh8t" Oct 30 17:40:07.881207 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:07.881174 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1c973-5d95487bd7-vmdzr" Oct 30 17:40:07.884687 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:07.884656 4965 generic.go:356] "Generic (PLEG): container finished" podID="32b71d3c-848d-453a-8a75-5b3e0281050d" containerID="f67e2fc7dcd3992c193c8c5616a6f952fe9ef47a55e8a062080c2e3268f885a1" exitCode=0 Oct 30 17:40:07.884905 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:07.884722 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d1c973-5d4f7449db-8dh8t" Oct 30 17:40:07.885334 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:07.884742 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d1c973-5d4f7449db-8dh8t" event={"ID":"32b71d3c-848d-453a-8a75-5b3e0281050d","Type":"ContainerDied","Data":"f67e2fc7dcd3992c193c8c5616a6f952fe9ef47a55e8a062080c2e3268f885a1"} Oct 30 17:40:07.885334 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:07.885127 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d1c973-5d4f7449db-8dh8t" event={"ID":"32b71d3c-848d-453a-8a75-5b3e0281050d","Type":"ContainerDied","Data":"f4d2b4d041d42fecc564cdcc70df89ad18695376eecc78cb75afe3251269c19a"} Oct 30 17:40:07.885334 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:07.885188 4965 scope.go:122] "RemoveContainer" containerID="f67e2fc7dcd3992c193c8c5616a6f952fe9ef47a55e8a062080c2e3268f885a1" Oct 30 17:40:07.887115 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:07.887091 4965 generic.go:356] "Generic (PLEG): container finished" podID="743b47bf-ff68-4a25-bb33-785d836c1cc9" containerID="82888a930a858928e9f0489be629e964206ce2fdb9d19bd258f4354f867f1f66" exitCode=0 Oct 30 17:40:07.887266 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:07.887249 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1c973-5d95487bd7-vmdzr" Oct 30 17:40:07.887492 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:07.887288 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1c973-5d95487bd7-vmdzr" event={"ID":"743b47bf-ff68-4a25-bb33-785d836c1cc9","Type":"ContainerDied","Data":"82888a930a858928e9f0489be629e964206ce2fdb9d19bd258f4354f867f1f66"} Oct 30 17:40:07.887576 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:07.887521 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1c973-5d95487bd7-vmdzr" event={"ID":"743b47bf-ff68-4a25-bb33-785d836c1cc9","Type":"ContainerDied","Data":"9e63c2a72ae275d37a4cd1fca650f4260be29464880fa5592c2a126a25018979"} Oct 30 17:40:07.902073 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:07.902049 4965 scope.go:122] "RemoveContainer" containerID="f67e2fc7dcd3992c193c8c5616a6f952fe9ef47a55e8a062080c2e3268f885a1" Oct 30 17:40:07.902434 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:40:07.902408 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f67e2fc7dcd3992c193c8c5616a6f952fe9ef47a55e8a062080c2e3268f885a1\": container with ID starting with f67e2fc7dcd3992c193c8c5616a6f952fe9ef47a55e8a062080c2e3268f885a1 not found: ID does not exist" containerID="f67e2fc7dcd3992c193c8c5616a6f952fe9ef47a55e8a062080c2e3268f885a1" Oct 30 17:40:07.902536 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:07.902442 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f67e2fc7dcd3992c193c8c5616a6f952fe9ef47a55e8a062080c2e3268f885a1"} err="failed to get container status \"f67e2fc7dcd3992c193c8c5616a6f952fe9ef47a55e8a062080c2e3268f885a1\": rpc error: code = NotFound desc = could not find container \"f67e2fc7dcd3992c193c8c5616a6f952fe9ef47a55e8a062080c2e3268f885a1\": container with ID starting with f67e2fc7dcd3992c193c8c5616a6f952fe9ef47a55e8a062080c2e3268f885a1 not found: ID does not exist" Oct 30 17:40:07.902536 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:07.902463 4965 scope.go:122] "RemoveContainer" containerID="82888a930a858928e9f0489be629e964206ce2fdb9d19bd258f4354f867f1f66" Oct 30 17:40:07.915360 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:07.915339 4965 scope.go:122] "RemoveContainer" containerID="82888a930a858928e9f0489be629e964206ce2fdb9d19bd258f4354f867f1f66" Oct 30 17:40:07.915782 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:40:07.915748 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82888a930a858928e9f0489be629e964206ce2fdb9d19bd258f4354f867f1f66\": container with ID starting with 82888a930a858928e9f0489be629e964206ce2fdb9d19bd258f4354f867f1f66 not found: ID does not exist" containerID="82888a930a858928e9f0489be629e964206ce2fdb9d19bd258f4354f867f1f66" Oct 30 17:40:07.915988 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:07.915803 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82888a930a858928e9f0489be629e964206ce2fdb9d19bd258f4354f867f1f66"} err="failed to get container status \"82888a930a858928e9f0489be629e964206ce2fdb9d19bd258f4354f867f1f66\": rpc error: code = NotFound desc = could not find container \"82888a930a858928e9f0489be629e964206ce2fdb9d19bd258f4354f867f1f66\": container with ID starting with 82888a930a858928e9f0489be629e964206ce2fdb9d19bd258f4354f867f1f66 not found: ID does not exist" Oct 30 17:40:07.932127 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:07.916568 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/743b47bf-ff68-4a25-bb33-785d836c1cc9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/743b47bf-ff68-4a25-bb33-785d836c1cc9-plugins-config-volume\") pod \"743b47bf-ff68-4a25-bb33-785d836c1cc9\" (UID: \"743b47bf-ff68-4a25-bb33-785d836c1cc9\") " Oct 30 17:40:07.932127 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:07.916745 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32b71d3c-848d-453a-8a75-5b3e0281050d-kube-api-access-v5pf2\" (UniqueName: \"kubernetes.io/projected/32b71d3c-848d-453a-8a75-5b3e0281050d-kube-api-access-v5pf2\") pod \"32b71d3c-848d-453a-8a75-5b3e0281050d\" (UID: \"32b71d3c-848d-453a-8a75-5b3e0281050d\") " Oct 30 17:40:07.932127 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:07.916975 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/743b47bf-ff68-4a25-bb33-785d836c1cc9-plugins-config-volume" pod "743b47bf-ff68-4a25-bb33-785d836c1cc9" (UID: "743b47bf-ff68-4a25-bb33-785d836c1cc9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:40:07.932127 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:07.917361 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32b71d3c-848d-453a-8a75-5b3e0281050d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32b71d3c-848d-453a-8a75-5b3e0281050d-plugins-config-volume\") pod \"32b71d3c-848d-453a-8a75-5b3e0281050d\" (UID: \"32b71d3c-848d-453a-8a75-5b3e0281050d\") " Oct 30 17:40:07.932127 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:07.917399 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/743b47bf-ff68-4a25-bb33-785d836c1cc9-kube-api-access-8kpdr\" (UniqueName: \"kubernetes.io/projected/743b47bf-ff68-4a25-bb33-785d836c1cc9-kube-api-access-8kpdr\") pod \"743b47bf-ff68-4a25-bb33-785d836c1cc9\" (UID: \"743b47bf-ff68-4a25-bb33-785d836c1cc9\") " Oct 30 17:40:07.932127 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:07.917851 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32b71d3c-848d-453a-8a75-5b3e0281050d-plugins-config-volume" pod "32b71d3c-848d-453a-8a75-5b3e0281050d" (UID: "32b71d3c-848d-453a-8a75-5b3e0281050d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:40:07.932127 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:07.918132 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/743b47bf-ff68-4a25-bb33-785d836c1cc9-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:40:07.932127 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:07.918154 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32b71d3c-848d-453a-8a75-5b3e0281050d-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:40:07.932127 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:07.932043 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/743b47bf-ff68-4a25-bb33-785d836c1cc9-kube-api-access-8kpdr" pod "743b47bf-ff68-4a25-bb33-785d836c1cc9" (UID: "743b47bf-ff68-4a25-bb33-785d836c1cc9"). InnerVolumeSpecName "kube-api-access-8kpdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:40:07.932689 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:07.932561 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32b71d3c-848d-453a-8a75-5b3e0281050d-kube-api-access-v5pf2" pod "32b71d3c-848d-453a-8a75-5b3e0281050d" (UID: "32b71d3c-848d-453a-8a75-5b3e0281050d"). InnerVolumeSpecName "kube-api-access-v5pf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:40:08.019533 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:08.019441 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kpdr\" (UniqueName: \"kubernetes.io/projected/743b47bf-ff68-4a25-bb33-785d836c1cc9-kube-api-access-8kpdr\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:40:08.019533 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:08.019471 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5pf2\" (UniqueName: \"kubernetes.io/projected/32b71d3c-848d-453a-8a75-5b3e0281050d-kube-api-access-v5pf2\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:40:08.228439 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:08.228403 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d1c973-5d4f7449db-8dh8t"] Oct 30 17:40:08.240250 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:08.240220 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d1c973-5d4f7449db-8dh8t"] Oct 30 17:40:08.266948 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:08.266912 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d1c973-5d95487bd7-vmdzr"] Oct 30 17:40:08.272919 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:08.272836 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d1c973-5d95487bd7-vmdzr"] Oct 30 17:40:08.900318 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:08.900255 4965 generic.go:356] "Generic (PLEG): container finished" podID="c7be7daf-3252-4355-99da-2cd6c664b3ae" containerID="408a5544a685adbd16ef188d221b11569357182c675be37b22f76d5be83094ca" exitCode=0 Oct 30 17:40:08.902652 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:08.900327 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" event={"ID":"c7be7daf-3252-4355-99da-2cd6c664b3ae","Type":"ContainerDied","Data":"408a5544a685adbd16ef188d221b11569357182c675be37b22f76d5be83094ca"} Oct 30 17:40:08.902652 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:08.900795 4965 scope.go:122] "RemoveContainer" containerID="408a5544a685adbd16ef188d221b11569357182c675be37b22f76d5be83094ca" Oct 30 17:40:08.903024 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:08.902856 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:40:08.903024 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:08.902957 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:40:08.903024 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:08.903004 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:40:09.910968 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:09.910926 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" event={"ID":"c7be7daf-3252-4355-99da-2cd6c664b3ae","Type":"ContainerStarted","Data":"239c0e755645f125174b47c2b8df61144fa464980db515f21a778838e3d567a4"} Oct 30 17:40:09.913251 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:09.913217 4965 generic.go:356] "Generic (PLEG): container finished" podID="30e31aef-c00b-4418-b005-50b3689d2912" containerID="73f7e236f32f6ded0e7e16ad8dffebf7b745fef40661d5a5517477120842b5f0" exitCode=0 Oct 30 17:40:09.913390 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:09.913291 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" event={"ID":"30e31aef-c00b-4418-b005-50b3689d2912","Type":"ContainerDied","Data":"73f7e236f32f6ded0e7e16ad8dffebf7b745fef40661d5a5517477120842b5f0"} Oct 30 17:40:09.913819 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:09.913802 4965 scope.go:122] "RemoveContainer" containerID="73f7e236f32f6ded0e7e16ad8dffebf7b745fef40661d5a5517477120842b5f0" Oct 30 17:40:09.916342 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:09.916295 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:40:09.916463 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:09.916384 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:40:09.916463 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:09.916435 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:40:10.120974 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:10.120935 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32b71d3c-848d-453a-8a75-5b3e0281050d" path="/var/lib/kubelet/pods/32b71d3c-848d-453a-8a75-5b3e0281050d/volumes" Oct 30 17:40:10.121498 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:10.121475 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="743b47bf-ff68-4a25-bb33-785d836c1cc9" path="/var/lib/kubelet/pods/743b47bf-ff68-4a25-bb33-785d836c1cc9/volumes" Oct 30 17:40:10.149086 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:10.147394 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:40:10.149086 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:10.147527 4965 patch_prober.go:29] interesting pod/e2e-api-a-802bda-openshift-default-664b986cf5-689jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 17:40:10.149086 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:10.147569 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" podUID="c7be7daf-3252-4355-99da-2cd6c664b3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 17:40:11.028884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:11.017629 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" event={"ID":"30e31aef-c00b-4418-b005-50b3689d2912","Type":"ContainerStarted","Data":"8e1f255523cf65c0a4396d77af9241f02a140f656c67ba7038fd92dacde165d0"} Oct 30 17:40:11.150313 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:11.149814 4965 patch_prober.go:29] interesting pod/e2e-api-a-802bda-openshift-default-664b986cf5-689jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 17:40:11.150313 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:11.149909 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" podUID="c7be7daf-3252-4355-99da-2cd6c664b3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 17:40:11.263890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:11.254671 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:40:11.263890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:11.254794 4965 patch_prober.go:29] interesting pod/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 17:40:11.263890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:11.254846 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" podUID="30e31aef-c00b-4418-b005-50b3689d2912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 17:40:12.144186 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:12.144150 4965 patch_prober.go:29] interesting pod/e2e-api-a-802bda-openshift-default-664b986cf5-689jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 17:40:12.144745 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:12.144212 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" podUID="c7be7daf-3252-4355-99da-2cd6c664b3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 17:40:12.239736 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:12.239697 4965 patch_prober.go:29] interesting pod/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 17:40:12.239949 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:12.239753 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" podUID="30e31aef-c00b-4418-b005-50b3689d2912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 17:40:13.158481 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:13.158442 4965 patch_prober.go:29] interesting pod/e2e-api-a-802bda-openshift-default-664b986cf5-689jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 17:40:13.168931 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:13.168493 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" podUID="c7be7daf-3252-4355-99da-2cd6c664b3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 17:40:13.245886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:13.239411 4965 patch_prober.go:29] interesting pod/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 17:40:13.245886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:13.239470 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" podUID="30e31aef-c00b-4418-b005-50b3689d2912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 17:40:14.144276 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:14.144239 4965 patch_prober.go:29] interesting pod/e2e-api-a-802bda-openshift-default-664b986cf5-689jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 17:40:14.144468 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:14.144297 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" podUID="c7be7daf-3252-4355-99da-2cd6c664b3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 17:40:14.239252 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:14.239217 4965 patch_prober.go:29] interesting pod/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 17:40:14.239650 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:14.239267 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" podUID="30e31aef-c00b-4418-b005-50b3689d2912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 17:40:15.146887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:15.146477 4965 patch_prober.go:29] interesting pod/e2e-api-a-802bda-openshift-default-664b986cf5-689jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 17:40:15.146887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:15.146537 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" podUID="c7be7daf-3252-4355-99da-2cd6c664b3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 17:40:15.239951 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:15.239914 4965 patch_prober.go:29] interesting pod/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 17:40:15.240336 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:15.239965 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" podUID="30e31aef-c00b-4418-b005-50b3689d2912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 17:40:16.144937 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:16.144856 4965 patch_prober.go:29] interesting pod/e2e-api-a-802bda-openshift-default-664b986cf5-689jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 17:40:16.145127 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:16.144993 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" podUID="c7be7daf-3252-4355-99da-2cd6c664b3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 17:40:16.239752 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:16.239715 4965 patch_prober.go:29] interesting pod/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 17:40:16.239954 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:16.239789 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" podUID="30e31aef-c00b-4418-b005-50b3689d2912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 17:40:17.218412 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:17.218377 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:40:17.240532 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:17.240494 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:40:17.242247 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:17.242215 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:40:17.261621 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:17.261593 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:40:17.621947 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:17.621908 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:40:18.339381 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:18.338224 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:40:22.488689 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:22.488652 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-49e155-5997c8f79-vqj2r"] Oct 30 17:40:22.489875 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:22.489843 4965 cpu_manager.go:414] "containerMap: removing container" podUID="a48121a5-ca41-4b77-a554-a08b069c97ea" containerName="payload-pre-processing" Oct 30 17:40:22.489950 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:22.489880 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a48121a5-ca41-4b77-a554-a08b069c97ea" containerName="payload-pre-processing" Oct 30 17:40:22.489993 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:22.489947 4965 cpu_manager.go:414] "containerMap: removing container" podUID="743b47bf-ff68-4a25-bb33-785d836c1cc9" containerName="payload-pre-processing" Oct 30 17:40:22.489993 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:22.489956 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="743b47bf-ff68-4a25-bb33-785d836c1cc9" containerName="payload-pre-processing" Oct 30 17:40:22.489993 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:22.489975 4965 cpu_manager.go:414] "containerMap: removing container" podUID="631b7466-f017-4430-89ee-f62b93d000bb" containerName="payload-pre-processing" Oct 30 17:40:22.489993 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:22.489983 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="631b7466-f017-4430-89ee-f62b93d000bb" containerName="payload-pre-processing" Oct 30 17:40:22.490115 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:22.490030 4965 cpu_manager.go:414] "containerMap: removing container" podUID="32b71d3c-848d-453a-8a75-5b3e0281050d" containerName="payload-processing" Oct 30 17:40:22.490115 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:22.490038 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32b71d3c-848d-453a-8a75-5b3e0281050d" containerName="payload-processing" Oct 30 17:40:22.490237 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:22.490214 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="743b47bf-ff68-4a25-bb33-785d836c1cc9" containerName="payload-pre-processing" Oct 30 17:40:22.490272 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:22.490244 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="631b7466-f017-4430-89ee-f62b93d000bb" containerName="payload-pre-processing" Oct 30 17:40:22.490272 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:22.490264 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="32b71d3c-848d-453a-8a75-5b3e0281050d" containerName="payload-processing" Oct 30 17:40:22.490335 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:22.490276 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="a48121a5-ca41-4b77-a554-a08b069c97ea" containerName="payload-pre-processing" Oct 30 17:40:22.495081 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:22.495057 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49e155-5997c8f79-vqj2r" Oct 30 17:40:22.528897 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:22.528845 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-49e155-5997c8f79-vqj2r"] Oct 30 17:40:22.551145 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:22.551110 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-49e155\"" Oct 30 17:40:22.585527 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:22.585492 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-49e155-dockercfg-mjkc4\"" Oct 30 17:40:22.614670 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:22.614630 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/112bfc33-9359-4f17-be86-f3ffd6ecae0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-49e155-5997c8f79-vqj2r\" (UID: \"112bfc33-9359-4f17-be86-f3ffd6ecae0f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-49e155-5997c8f79-vqj2r" Oct 30 17:40:22.615151 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:22.615105 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxwvt\" (UniqueName: \"kubernetes.io/projected/112bfc33-9359-4f17-be86-f3ffd6ecae0f-kube-api-access-dxwvt\") pod \"payload-pre-processing-e2e-api-b-49e155-5997c8f79-vqj2r\" (UID: \"112bfc33-9359-4f17-be86-f3ffd6ecae0f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-49e155-5997c8f79-vqj2r" Oct 30 17:40:22.782086 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:22.780068 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/112bfc33-9359-4f17-be86-f3ffd6ecae0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-49e155-5997c8f79-vqj2r\" (UID: \"112bfc33-9359-4f17-be86-f3ffd6ecae0f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-49e155-5997c8f79-vqj2r" Oct 30 17:40:22.782086 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:22.780286 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxwvt\" (UniqueName: \"kubernetes.io/projected/112bfc33-9359-4f17-be86-f3ffd6ecae0f-kube-api-access-dxwvt\") pod \"payload-pre-processing-e2e-api-b-49e155-5997c8f79-vqj2r\" (UID: \"112bfc33-9359-4f17-be86-f3ffd6ecae0f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-49e155-5997c8f79-vqj2r" Oct 30 17:40:22.782086 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:22.781776 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/112bfc33-9359-4f17-be86-f3ffd6ecae0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-49e155-5997c8f79-vqj2r\" (UID: \"112bfc33-9359-4f17-be86-f3ffd6ecae0f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-49e155-5997c8f79-vqj2r" Oct 30 17:40:22.953236 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:22.953207 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxwvt\" (UniqueName: \"kubernetes.io/projected/112bfc33-9359-4f17-be86-f3ffd6ecae0f-kube-api-access-dxwvt\") pod \"payload-pre-processing-e2e-api-b-49e155-5997c8f79-vqj2r\" (UID: \"112bfc33-9359-4f17-be86-f3ffd6ecae0f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-49e155-5997c8f79-vqj2r" Oct 30 17:40:23.111075 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:23.111044 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49e155-5997c8f79-vqj2r" Oct 30 17:40:23.424661 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:40:23.424625 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod112bfc33_9359_4f17_be86_f3ffd6ecae0f.slice/crio-bbfab169275c313eae811bc9ad11f17676baaec4e825e008c4248f463165fb25 WatchSource:0}: Error finding container bbfab169275c313eae811bc9ad11f17676baaec4e825e008c4248f463165fb25: Status 404 returned error can't find the container with id bbfab169275c313eae811bc9ad11f17676baaec4e825e008c4248f463165fb25 Oct 30 17:40:23.424830 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:23.424785 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-49e155-5997c8f79-vqj2r"] Oct 30 17:40:24.296066 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:24.296028 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cc4e96-758d6bddb4-zvx88"] Oct 30 17:40:24.301272 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:24.301248 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cc4e96-758d6bddb4-zvx88" Oct 30 17:40:24.321042 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:24.320977 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cc4e96-758d6bddb4-zvx88"] Oct 30 17:40:24.334782 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:24.327969 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-cc4e96\"" Oct 30 17:40:24.353097 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:24.353060 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49e155-5997c8f79-vqj2r" event={"ID":"112bfc33-9359-4f17-be86-f3ffd6ecae0f","Type":"ContainerStarted","Data":"4669f588933210f6b5779f12187f150405915dd20395c6f7c38d3e7bb290ba09"} Oct 30 17:40:24.353291 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:24.353118 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49e155-5997c8f79-vqj2r" event={"ID":"112bfc33-9359-4f17-be86-f3ffd6ecae0f","Type":"ContainerStarted","Data":"bbfab169275c313eae811bc9ad11f17676baaec4e825e008c4248f463165fb25"} Oct 30 17:40:24.353291 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:24.353148 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49e155-5997c8f79-vqj2r" Oct 30 17:40:24.401753 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:24.401685 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49e155-5997c8f79-vqj2r" podStartSLOduration=2.401666211 podStartE2EDuration="2.401666211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:40:24.401252989 +0000 UTC m=+1682.453352944" watchObservedRunningTime="2026-10-30 17:40:24.401666211 +0000 UTC m=+1682.453766143" Oct 30 17:40:24.412164 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:24.412135 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh6l9\" (UniqueName: \"kubernetes.io/projected/7be33b0f-8e47-4fcc-a091-8aaec48cd6bd-kube-api-access-jh6l9\") pod \"payload-processing-e2e-api-b-cc4e96-758d6bddb4-zvx88\" (UID: \"7be33b0f-8e47-4fcc-a091-8aaec48cd6bd\") " pod="openshift-ingress/payload-processing-e2e-api-b-cc4e96-758d6bddb4-zvx88" Oct 30 17:40:24.412397 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:24.412368 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7be33b0f-8e47-4fcc-a091-8aaec48cd6bd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cc4e96-758d6bddb4-zvx88\" (UID: \"7be33b0f-8e47-4fcc-a091-8aaec48cd6bd\") " pod="openshift-ingress/payload-processing-e2e-api-b-cc4e96-758d6bddb4-zvx88" Oct 30 17:40:24.513957 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:24.513916 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh6l9\" (UniqueName: \"kubernetes.io/projected/7be33b0f-8e47-4fcc-a091-8aaec48cd6bd-kube-api-access-jh6l9\") pod \"payload-processing-e2e-api-b-cc4e96-758d6bddb4-zvx88\" (UID: \"7be33b0f-8e47-4fcc-a091-8aaec48cd6bd\") " pod="openshift-ingress/payload-processing-e2e-api-b-cc4e96-758d6bddb4-zvx88" Oct 30 17:40:24.514147 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:24.514096 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7be33b0f-8e47-4fcc-a091-8aaec48cd6bd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cc4e96-758d6bddb4-zvx88\" (UID: \"7be33b0f-8e47-4fcc-a091-8aaec48cd6bd\") " pod="openshift-ingress/payload-processing-e2e-api-b-cc4e96-758d6bddb4-zvx88" Oct 30 17:40:24.514797 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:24.514775 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7be33b0f-8e47-4fcc-a091-8aaec48cd6bd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cc4e96-758d6bddb4-zvx88\" (UID: \"7be33b0f-8e47-4fcc-a091-8aaec48cd6bd\") " pod="openshift-ingress/payload-processing-e2e-api-b-cc4e96-758d6bddb4-zvx88" Oct 30 17:40:24.537349 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:24.537322 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh6l9\" (UniqueName: \"kubernetes.io/projected/7be33b0f-8e47-4fcc-a091-8aaec48cd6bd-kube-api-access-jh6l9\") pod \"payload-processing-e2e-api-b-cc4e96-758d6bddb4-zvx88\" (UID: \"7be33b0f-8e47-4fcc-a091-8aaec48cd6bd\") " pod="openshift-ingress/payload-processing-e2e-api-b-cc4e96-758d6bddb4-zvx88" Oct 30 17:40:24.617844 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:24.617810 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cc4e96-758d6bddb4-zvx88" Oct 30 17:40:25.428370 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:25.428335 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cc4e96-758d6bddb4-zvx88"] Oct 30 17:40:25.432165 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:40:25.432134 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7be33b0f_8e47_4fcc_a091_8aaec48cd6bd.slice/crio-4dffb43a223f1bb89a95d4dc6617316859d503f7823cee6c836eeea68a827bf7 WatchSource:0}: Error finding container 4dffb43a223f1bb89a95d4dc6617316859d503f7823cee6c836eeea68a827bf7: Status 404 returned error can't find the container with id 4dffb43a223f1bb89a95d4dc6617316859d503f7823cee6c836eeea68a827bf7 Oct 30 17:40:26.380384 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:26.380346 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cc4e96-758d6bddb4-zvx88" event={"ID":"7be33b0f-8e47-4fcc-a091-8aaec48cd6bd","Type":"ContainerStarted","Data":"fae2241323d75bd4f5e018be322fce01bcda07785f89d6d10a9d0cd7e77e1365"} Oct 30 17:40:26.380562 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:26.380402 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cc4e96-758d6bddb4-zvx88" event={"ID":"7be33b0f-8e47-4fcc-a091-8aaec48cd6bd","Type":"ContainerStarted","Data":"4dffb43a223f1bb89a95d4dc6617316859d503f7823cee6c836eeea68a827bf7"} Oct 30 17:40:26.380562 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:26.380423 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-cc4e96-758d6bddb4-zvx88" Oct 30 17:40:26.422605 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:26.422545 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-cc4e96-758d6bddb4-zvx88" podStartSLOduration=2.422526499 podStartE2EDuration="2.422526499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:40:26.419608759 +0000 UTC m=+1684.471708711" watchObservedRunningTime="2026-10-30 17:40:26.422526499 +0000 UTC m=+1684.474626431" Oct 30 17:40:34.568347 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:34.568309 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:40:35.375901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:35.375853 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49e155-5997c8f79-vqj2r" Oct 30 17:40:35.998063 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:40:35.998017 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:40:37.391733 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:40:37.391707 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-cc4e96-758d6bddb4-zvx88" Oct 30 17:41:06.074832 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:41:06.074796 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:41:08.588120 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:08.588087 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k"] Oct 30 17:41:09.284125 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.284089 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7"] Oct 30 17:41:09.303942 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.291903 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:09.306921 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.306844 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7"] Oct 30 17:41:09.446561 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.446529 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-workload-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-gxts7\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:09.446737 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.446649 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-istio-data\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-gxts7\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:09.446737 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.446719 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09efdc53-164b-4276-8179-5dffd1c33896-istiod-ca-cert\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-gxts7\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:09.446830 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.446817 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09efdc53-164b-4276-8179-5dffd1c33896-istio-token\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-gxts7\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:09.446929 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.446882 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-istio-envoy\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-gxts7\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:09.447044 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.447024 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-workload-certs\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-gxts7\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:09.447084 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.447053 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhdm5\" (UniqueName: \"kubernetes.io/projected/09efdc53-164b-4276-8179-5dffd1c33896-kube-api-access-qhdm5\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-gxts7\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:09.447185 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.447168 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-credential-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-gxts7\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:09.447227 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.447200 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09efdc53-164b-4276-8179-5dffd1c33896-istio-podinfo\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-gxts7\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:09.549035 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.548949 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-workload-certs\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-gxts7\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:09.549035 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.548985 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhdm5\" (UniqueName: \"kubernetes.io/projected/09efdc53-164b-4276-8179-5dffd1c33896-kube-api-access-qhdm5\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-gxts7\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:09.549281 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.549047 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-credential-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-gxts7\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:09.549281 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.549069 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09efdc53-164b-4276-8179-5dffd1c33896-istio-podinfo\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-gxts7\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:09.549281 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.549127 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-workload-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-gxts7\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:09.549281 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.549222 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-istio-data\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-gxts7\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:09.549513 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.549293 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09efdc53-164b-4276-8179-5dffd1c33896-istiod-ca-cert\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-gxts7\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:09.549513 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.549371 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09efdc53-164b-4276-8179-5dffd1c33896-istio-token\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-gxts7\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:09.549513 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.549436 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-workload-certs\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-gxts7\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:09.549513 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.549446 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-istio-envoy\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-gxts7\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:09.549513 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.549489 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-credential-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-gxts7\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:09.549736 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.549546 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-workload-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-gxts7\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:09.549736 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.549715 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-istio-data\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-gxts7\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:09.550086 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.550066 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09efdc53-164b-4276-8179-5dffd1c33896-istiod-ca-cert\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-gxts7\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:09.551579 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.551555 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-istio-envoy\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-gxts7\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:09.551698 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.551667 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09efdc53-164b-4276-8179-5dffd1c33896-istio-podinfo\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-gxts7\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:09.572747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.572725 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09efdc53-164b-4276-8179-5dffd1c33896-istio-token\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-gxts7\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:09.572930 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.572907 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhdm5\" (UniqueName: \"kubernetes.io/projected/09efdc53-164b-4276-8179-5dffd1c33896-kube-api-access-qhdm5\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-gxts7\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:09.630085 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.630060 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:09.954109 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:41:09.954073 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09efdc53_164b_4276_8179_5dffd1c33896.slice/crio-2a13a346b0392533f804be12cdb95c7fffad438052b80ac7ecc157463c556e87 WatchSource:0}: Error finding container 2a13a346b0392533f804be12cdb95c7fffad438052b80ac7ecc157463c556e87: Status 404 returned error can't find the container with id 2a13a346b0392533f804be12cdb95c7fffad438052b80ac7ecc157463c556e87 Oct 30 17:41:09.954785 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.954758 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7"] Oct 30 17:41:09.967221 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.958416 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:41:09.967221 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.958687 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:41:09.967221 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:09.958782 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:41:10.880091 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:10.880054 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" event={"ID":"09efdc53-164b-4276-8179-5dffd1c33896","Type":"ContainerStarted","Data":"7d7b88394f872a7206a1d498246fda0b5a7e3db3b6b088b86147cc1d3fb76328"} Oct 30 17:41:10.880516 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:10.880108 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" event={"ID":"09efdc53-164b-4276-8179-5dffd1c33896","Type":"ContainerStarted","Data":"2a13a346b0392533f804be12cdb95c7fffad438052b80ac7ecc157463c556e87"} Oct 30 17:41:10.938955 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:10.938856 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" podStartSLOduration=1.938841324 podStartE2EDuration="1.938841324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:41:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:41:10.93608669 +0000 UTC m=+1728.988186621" watchObservedRunningTime="2026-10-30 17:41:10.938841324 +0000 UTC m=+1728.990941302" Oct 30 17:41:11.631324 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:11.631286 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:11.633191 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:11.633165 4965 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 17:41:11.633344 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:11.633209 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" podUID="09efdc53-164b-4276-8179-5dffd1c33896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 17:41:12.630408 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:12.630375 4965 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 17:41:12.631076 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:12.630433 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" podUID="09efdc53-164b-4276-8179-5dffd1c33896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 17:41:13.630629 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:13.630588 4965 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 17:41:13.631169 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:13.630650 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" podUID="09efdc53-164b-4276-8179-5dffd1c33896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 17:41:14.631316 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:14.631279 4965 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 17:41:14.631910 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:14.631876 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" podUID="09efdc53-164b-4276-8179-5dffd1c33896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 17:41:15.631386 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:15.631350 4965 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 17:41:15.631794 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:15.631415 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" podUID="09efdc53-164b-4276-8179-5dffd1c33896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 17:41:16.631218 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:16.631186 4965 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 17:41:16.631411 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:16.631244 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" podUID="09efdc53-164b-4276-8179-5dffd1c33896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 17:41:17.635251 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:17.635222 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:17.635693 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:17.635673 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:17.636327 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:17.636304 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:17.987617 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:17.987526 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc"] Oct 30 17:41:17.987829 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:17.987795 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc" podUID="936c85c5-969e-47f9-b58f-57af5a0d36d9" containerName="maas-api" containerID="cri-o://ab74cd7b3135112ba3e8df91ec3c42c226866cba3eefb4834eeb0d77d7b682a1" gracePeriod=30 Oct 30 17:41:18.476616 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:18.476267 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc" Oct 30 17:41:18.585407 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:18.577595 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/936c85c5-969e-47f9-b58f-57af5a0d36d9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/936c85c5-969e-47f9-b58f-57af5a0d36d9-maas-api-tls\") pod \"936c85c5-969e-47f9-b58f-57af5a0d36d9\" (UID: \"936c85c5-969e-47f9-b58f-57af5a0d36d9\") " Oct 30 17:41:18.585407 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:18.577897 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/936c85c5-969e-47f9-b58f-57af5a0d36d9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/936c85c5-969e-47f9-b58f-57af5a0d36d9-openshift-service-ca\") pod \"936c85c5-969e-47f9-b58f-57af5a0d36d9\" (UID: \"936c85c5-969e-47f9-b58f-57af5a0d36d9\") " Oct 30 17:41:18.585407 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:18.577997 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/936c85c5-969e-47f9-b58f-57af5a0d36d9-kube-api-access-k2zg7\" (UniqueName: \"kubernetes.io/projected/936c85c5-969e-47f9-b58f-57af5a0d36d9-kube-api-access-k2zg7\") pod \"936c85c5-969e-47f9-b58f-57af5a0d36d9\" (UID: \"936c85c5-969e-47f9-b58f-57af5a0d36d9\") " Oct 30 17:41:18.585407 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:18.578085 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/936c85c5-969e-47f9-b58f-57af5a0d36d9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/936c85c5-969e-47f9-b58f-57af5a0d36d9-metrics-certs\") pod \"936c85c5-969e-47f9-b58f-57af5a0d36d9\" (UID: \"936c85c5-969e-47f9-b58f-57af5a0d36d9\") " Oct 30 17:41:18.597700 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:18.597650 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/936c85c5-969e-47f9-b58f-57af5a0d36d9-openshift-service-ca" pod "936c85c5-969e-47f9-b58f-57af5a0d36d9" (UID: "936c85c5-969e-47f9-b58f-57af5a0d36d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:41:18.794179 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:18.794148 4965 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/936c85c5-969e-47f9-b58f-57af5a0d36d9-openshift-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:18.839310 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:18.839222 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/936c85c5-969e-47f9-b58f-57af5a0d36d9-maas-api-tls" pod "936c85c5-969e-47f9-b58f-57af5a0d36d9" (UID: "936c85c5-969e-47f9-b58f-57af5a0d36d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:41:18.843499 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:18.843432 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/936c85c5-969e-47f9-b58f-57af5a0d36d9-kube-api-access-k2zg7" pod "936c85c5-969e-47f9-b58f-57af5a0d36d9" (UID: "936c85c5-969e-47f9-b58f-57af5a0d36d9"). InnerVolumeSpecName "kube-api-access-k2zg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:18.846280 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:18.846233 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/936c85c5-969e-47f9-b58f-57af5a0d36d9-metrics-certs" pod "936c85c5-969e-47f9-b58f-57af5a0d36d9" (UID: "936c85c5-969e-47f9-b58f-57af5a0d36d9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:41:18.907605 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:18.907567 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/936c85c5-969e-47f9-b58f-57af5a0d36d9-maas-api-tls\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:18.907855 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:18.907838 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2zg7\" (UniqueName: \"kubernetes.io/projected/936c85c5-969e-47f9-b58f-57af5a0d36d9-kube-api-access-k2zg7\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:18.907994 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:18.907976 4965 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/936c85c5-969e-47f9-b58f-57af5a0d36d9-metrics-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:18.988833 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:18.988799 4965 generic.go:356] "Generic (PLEG): container finished" podID="936c85c5-969e-47f9-b58f-57af5a0d36d9" containerID="ab74cd7b3135112ba3e8df91ec3c42c226866cba3eefb4834eeb0d77d7b682a1" exitCode=0 Oct 30 17:41:18.989012 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:18.988854 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc" Oct 30 17:41:18.989012 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:18.988884 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc" event={"ID":"936c85c5-969e-47f9-b58f-57af5a0d36d9","Type":"ContainerDied","Data":"ab74cd7b3135112ba3e8df91ec3c42c226866cba3eefb4834eeb0d77d7b682a1"} Oct 30 17:41:18.989087 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:18.989041 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc" event={"ID":"936c85c5-969e-47f9-b58f-57af5a0d36d9","Type":"ContainerDied","Data":"3d1c60f491f55f4059dd3e31deff7f44a207298db4ef40637e77ec1c2c82b57a"} Oct 30 17:41:18.989087 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:18.989066 4965 scope.go:122] "RemoveContainer" containerID="ab74cd7b3135112ba3e8df91ec3c42c226866cba3eefb4834eeb0d77d7b682a1" Oct 30 17:41:19.005021 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:19.004996 4965 scope.go:122] "RemoveContainer" containerID="ab74cd7b3135112ba3e8df91ec3c42c226866cba3eefb4834eeb0d77d7b682a1" Oct 30 17:41:19.005438 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:41:19.005408 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab74cd7b3135112ba3e8df91ec3c42c226866cba3eefb4834eeb0d77d7b682a1\": container with ID starting with ab74cd7b3135112ba3e8df91ec3c42c226866cba3eefb4834eeb0d77d7b682a1 not found: ID does not exist" containerID="ab74cd7b3135112ba3e8df91ec3c42c226866cba3eefb4834eeb0d77d7b682a1" Oct 30 17:41:19.005560 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:19.005442 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab74cd7b3135112ba3e8df91ec3c42c226866cba3eefb4834eeb0d77d7b682a1"} err="failed to get container status \"ab74cd7b3135112ba3e8df91ec3c42c226866cba3eefb4834eeb0d77d7b682a1\": rpc error: code = NotFound desc = could not find container \"ab74cd7b3135112ba3e8df91ec3c42c226866cba3eefb4834eeb0d77d7b682a1\": container with ID starting with ab74cd7b3135112ba3e8df91ec3c42c226866cba3eefb4834eeb0d77d7b682a1 not found: ID does not exist" Oct 30 17:41:19.044645 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:19.044612 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc"] Oct 30 17:41:19.052881 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:41:19.052841 4965 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod936c85c5_969e_47f9_b58f_57af5a0d36d9.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod936c85c5_969e_47f9_b58f_57af5a0d36d9.slice/crio-3d1c60f491f55f4059dd3e31deff7f44a207298db4ef40637e77ec1c2c82b57a\": RecentStats: unable to find data in memory cache]" Oct 30 17:41:19.056356 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:19.056333 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d1c973-8669f7d986-gbwpc"] Oct 30 17:41:19.588141 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:19.588100 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:41:20.101338 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:20.101304 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="936c85c5-969e-47f9-b58f-57af5a0d36d9" path="/var/lib/kubelet/pods/936c85c5-969e-47f9-b58f-57af5a0d36d9/volumes" Oct 30 17:41:23.015995 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:23.015941 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:41:23.021855 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:23.021823 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rh774 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:41:23.022097 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:23.022067 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" podUID="337938ce-7eb2-45bf-a5aa-0ef36cee9444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:41:23.068985 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:23.068949 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-zv8m5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:41:23.069258 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:23.069226 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" podUID="0a692092-9a09-4a93-b285-675f8268eb68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:41:23.157999 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:23.157958 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:41:23.485883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:23.484974 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:41:23.647881 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:23.641925 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" podUID="7715de9f-65c1-44fb-b83a-692ec07b30a8" containerName="main" containerID="cri-o://c3e8dfc4684e450367e82d52bed9b9ceb0c6722825dce3da857d0ccd1a2ec854" gracePeriod=45 Oct 30 17:41:23.731264 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:23.731224 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" podUID="7715de9f-65c1-44fb-b83a-692ec07b30a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": EOF" Oct 30 17:41:24.029186 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.029156 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" Oct 30 17:41:24.129509 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.129469 4965 generic.go:356] "Generic (PLEG): container finished" podID="7715de9f-65c1-44fb-b83a-692ec07b30a8" containerID="c3e8dfc4684e450367e82d52bed9b9ceb0c6722825dce3da857d0ccd1a2ec854" exitCode=0 Oct 30 17:41:24.130397 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.129560 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" Oct 30 17:41:24.130397 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.129689 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" event={"ID":"7715de9f-65c1-44fb-b83a-692ec07b30a8","Type":"ContainerDied","Data":"c3e8dfc4684e450367e82d52bed9b9ceb0c6722825dce3da857d0ccd1a2ec854"} Oct 30 17:41:24.130397 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.129802 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k" event={"ID":"7715de9f-65c1-44fb-b83a-692ec07b30a8","Type":"ContainerDied","Data":"30d6246d4d38905ab61df06b82fe39ef147d336df57fbfbe648a27571ca5d1f2"} Oct 30 17:41:24.130397 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.129839 4965 scope.go:122] "RemoveContainer" containerID="c3e8dfc4684e450367e82d52bed9b9ceb0c6722825dce3da857d0ccd1a2ec854" Oct 30 17:41:24.149816 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.149794 4965 scope.go:122] "RemoveContainer" containerID="c3e8dfc4684e450367e82d52bed9b9ceb0c6722825dce3da857d0ccd1a2ec854" Oct 30 17:41:24.150137 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:41:24.150110 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3e8dfc4684e450367e82d52bed9b9ceb0c6722825dce3da857d0ccd1a2ec854\": container with ID starting with c3e8dfc4684e450367e82d52bed9b9ceb0c6722825dce3da857d0ccd1a2ec854 not found: ID does not exist" containerID="c3e8dfc4684e450367e82d52bed9b9ceb0c6722825dce3da857d0ccd1a2ec854" Oct 30 17:41:24.150281 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.150150 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3e8dfc4684e450367e82d52bed9b9ceb0c6722825dce3da857d0ccd1a2ec854"} err="failed to get container status \"c3e8dfc4684e450367e82d52bed9b9ceb0c6722825dce3da857d0ccd1a2ec854\": rpc error: code = NotFound desc = could not find container \"c3e8dfc4684e450367e82d52bed9b9ceb0c6722825dce3da857d0ccd1a2ec854\": container with ID starting with c3e8dfc4684e450367e82d52bed9b9ceb0c6722825dce3da857d0ccd1a2ec854 not found: ID does not exist" Oct 30 17:41:24.151910 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.151833 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7715de9f-65c1-44fb-b83a-692ec07b30a8-tls-certs\" (UniqueName: \"kubernetes.io/secret/7715de9f-65c1-44fb-b83a-692ec07b30a8-tls-certs\") pod \"7715de9f-65c1-44fb-b83a-692ec07b30a8\" (UID: \"7715de9f-65c1-44fb-b83a-692ec07b30a8\") " Oct 30 17:41:24.152017 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.151942 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7715de9f-65c1-44fb-b83a-692ec07b30a8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7715de9f-65c1-44fb-b83a-692ec07b30a8-dshm\") pod \"7715de9f-65c1-44fb-b83a-692ec07b30a8\" (UID: \"7715de9f-65c1-44fb-b83a-692ec07b30a8\") " Oct 30 17:41:24.152017 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.151977 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7715de9f-65c1-44fb-b83a-692ec07b30a8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7715de9f-65c1-44fb-b83a-692ec07b30a8-model-cache\") pod \"7715de9f-65c1-44fb-b83a-692ec07b30a8\" (UID: \"7715de9f-65c1-44fb-b83a-692ec07b30a8\") " Oct 30 17:41:24.152209 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.152048 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7715de9f-65c1-44fb-b83a-692ec07b30a8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7715de9f-65c1-44fb-b83a-692ec07b30a8-tmp-dir\") pod \"7715de9f-65c1-44fb-b83a-692ec07b30a8\" (UID: \"7715de9f-65c1-44fb-b83a-692ec07b30a8\") " Oct 30 17:41:24.152209 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.152149 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7715de9f-65c1-44fb-b83a-692ec07b30a8-kube-api-access-59qfj\" (UniqueName: \"kubernetes.io/projected/7715de9f-65c1-44fb-b83a-692ec07b30a8-kube-api-access-59qfj\") pod \"7715de9f-65c1-44fb-b83a-692ec07b30a8\" (UID: \"7715de9f-65c1-44fb-b83a-692ec07b30a8\") " Oct 30 17:41:24.152209 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.152178 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7715de9f-65c1-44fb-b83a-692ec07b30a8-home\" (UniqueName: \"kubernetes.io/empty-dir/7715de9f-65c1-44fb-b83a-692ec07b30a8-home\") pod \"7715de9f-65c1-44fb-b83a-692ec07b30a8\" (UID: \"7715de9f-65c1-44fb-b83a-692ec07b30a8\") " Oct 30 17:41:24.152556 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.152533 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7715de9f-65c1-44fb-b83a-692ec07b30a8-tmp-dir" pod "7715de9f-65c1-44fb-b83a-692ec07b30a8" (UID: "7715de9f-65c1-44fb-b83a-692ec07b30a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:24.152825 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.152797 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7715de9f-65c1-44fb-b83a-692ec07b30a8-model-cache" pod "7715de9f-65c1-44fb-b83a-692ec07b30a8" (UID: "7715de9f-65c1-44fb-b83a-692ec07b30a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:24.153236 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.153197 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7715de9f-65c1-44fb-b83a-692ec07b30a8-home" pod "7715de9f-65c1-44fb-b83a-692ec07b30a8" (UID: "7715de9f-65c1-44fb-b83a-692ec07b30a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:24.153530 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.153501 4965 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7715de9f-65c1-44fb-b83a-692ec07b30a8-home\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:24.153633 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.153533 4965 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7715de9f-65c1-44fb-b83a-692ec07b30a8-model-cache\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:24.153633 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.153549 4965 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7715de9f-65c1-44fb-b83a-692ec07b30a8-tmp-dir\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:24.156804 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.156777 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7715de9f-65c1-44fb-b83a-692ec07b30a8-kube-api-access-59qfj" pod "7715de9f-65c1-44fb-b83a-692ec07b30a8" (UID: "7715de9f-65c1-44fb-b83a-692ec07b30a8"). InnerVolumeSpecName "kube-api-access-59qfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:24.160296 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.160266 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7715de9f-65c1-44fb-b83a-692ec07b30a8-dshm" pod "7715de9f-65c1-44fb-b83a-692ec07b30a8" (UID: "7715de9f-65c1-44fb-b83a-692ec07b30a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:24.160401 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.160312 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7715de9f-65c1-44fb-b83a-692ec07b30a8-tls-certs" pod "7715de9f-65c1-44fb-b83a-692ec07b30a8" (UID: "7715de9f-65c1-44fb-b83a-692ec07b30a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:41:24.255176 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.255082 4965 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7715de9f-65c1-44fb-b83a-692ec07b30a8-tls-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:24.255176 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.255116 4965 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7715de9f-65c1-44fb-b83a-692ec07b30a8-dshm\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:24.255176 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.255129 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59qfj\" (UniqueName: \"kubernetes.io/projected/7715de9f-65c1-44fb-b83a-692ec07b30a8-kube-api-access-59qfj\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:24.328491 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.328456 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57"] Oct 30 17:41:24.329387 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.329365 4965 cpu_manager.go:414] "containerMap: removing container" podUID="7715de9f-65c1-44fb-b83a-692ec07b30a8" containerName="main" Oct 30 17:41:24.329387 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.329385 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7715de9f-65c1-44fb-b83a-692ec07b30a8" containerName="main" Oct 30 17:41:24.329536 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.329418 4965 cpu_manager.go:414] "containerMap: removing container" podUID="936c85c5-969e-47f9-b58f-57af5a0d36d9" containerName="maas-api" Oct 30 17:41:24.329536 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.329423 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="936c85c5-969e-47f9-b58f-57af5a0d36d9" containerName="maas-api" Oct 30 17:41:24.329605 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.329589 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="7715de9f-65c1-44fb-b83a-692ec07b30a8" containerName="main" Oct 30 17:41:24.329605 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.329598 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="936c85c5-969e-47f9-b58f-57af5a0d36d9" containerName="maas-api" Oct 30 17:41:24.335025 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.334997 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:24.356132 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.346975 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp"] Oct 30 17:41:24.377926 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.366383 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57"] Oct 30 17:41:24.377926 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.366509 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:24.377926 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.375700 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp"] Oct 30 17:41:24.460754 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.460719 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-workload-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:24.460964 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.460771 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-workload-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-r6v57\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:24.460964 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.460833 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz2bq\" (UniqueName: \"kubernetes.io/projected/a4c7171d-f08d-484e-8201-2e8644360ede-kube-api-access-cz2bq\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-r6v57\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:24.460964 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.460906 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-istio-data\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-r6v57\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:24.460964 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.460941 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-workload-certs\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:24.461140 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.460970 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istio-envoy\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:24.461140 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.461076 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4c7171d-f08d-484e-8201-2e8644360ede-istiod-ca-cert\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-r6v57\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:24.461140 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.461104 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4c7171d-f08d-484e-8201-2e8644360ede-istio-token\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-r6v57\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:24.461238 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.461210 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-credential-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:24.461278 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.461237 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istio-data\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:24.461278 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.461270 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49k48\" (UniqueName: \"kubernetes.io/projected/123ee2af-0589-49d5-b1a5-3f68b6ac9432-kube-api-access-49k48\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:24.461347 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.461316 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4c7171d-f08d-484e-8201-2e8644360ede-istio-podinfo\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-r6v57\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:24.461421 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.461407 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istio-token\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:24.461458 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.461430 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istio-podinfo\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:24.461498 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.461473 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istiod-ca-cert\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:24.461535 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.461509 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-credential-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-r6v57\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:24.461576 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.461539 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-istio-envoy\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-r6v57\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:24.461638 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.461623 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-workload-certs\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-r6v57\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:24.503973 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.503943 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k"] Oct 30 17:41:24.531992 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.531920 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-81e9336e/e2e-lifecycle-model-81e9336e-kserve-57888489d9-j2j9k"] Oct 30 17:41:24.562905 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.562851 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-credential-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:24.563072 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.562907 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istio-data\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:24.563072 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.562951 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49k48\" (UniqueName: \"kubernetes.io/projected/123ee2af-0589-49d5-b1a5-3f68b6ac9432-kube-api-access-49k48\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:24.563072 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.562975 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4c7171d-f08d-484e-8201-2e8644360ede-istio-podinfo\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-r6v57\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:24.563072 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.563029 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istio-token\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:24.563238 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.563073 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istio-podinfo\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:24.563238 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.563151 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istiod-ca-cert\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:24.563238 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.563187 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-credential-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-r6v57\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:24.563238 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.563212 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-istio-envoy\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-r6v57\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:24.563442 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.563281 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-credential-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:24.563442 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.563313 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-workload-certs\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-r6v57\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:24.563442 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.563326 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istio-data\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:24.563442 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.563397 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-workload-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:24.563663 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.563456 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-workload-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-r6v57\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:24.563663 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.563494 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz2bq\" (UniqueName: \"kubernetes.io/projected/a4c7171d-f08d-484e-8201-2e8644360ede-kube-api-access-cz2bq\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-r6v57\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:24.563663 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.563522 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-istio-data\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-r6v57\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:24.563663 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.563627 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-credential-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-r6v57\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:24.563663 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.563633 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-workload-certs\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:24.563960 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.563692 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-workload-certs\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-r6v57\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:24.563960 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.563713 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-workload-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:24.563960 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.563719 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istio-envoy\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:24.563960 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.563803 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4c7171d-f08d-484e-8201-2e8644360ede-istiod-ca-cert\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-r6v57\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:24.563960 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.563848 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4c7171d-f08d-484e-8201-2e8644360ede-istio-token\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-r6v57\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:24.563960 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.563949 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-workload-certs\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:24.564271 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.564205 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-istio-data\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-r6v57\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:24.564415 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.564396 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-workload-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-r6v57\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:24.564939 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.564917 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4c7171d-f08d-484e-8201-2e8644360ede-istiod-ca-cert\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-r6v57\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:24.565003 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.564962 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istiod-ca-cert\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:24.565991 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.565969 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istio-envoy\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:24.566086 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.566066 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4c7171d-f08d-484e-8201-2e8644360ede-istio-podinfo\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-r6v57\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:24.566130 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.566104 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-istio-envoy\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-r6v57\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:24.566330 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.566312 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istio-podinfo\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:24.587269 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.587242 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istio-token\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:24.587387 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.587311 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49k48\" (UniqueName: \"kubernetes.io/projected/123ee2af-0589-49d5-b1a5-3f68b6ac9432-kube-api-access-49k48\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:24.588514 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.588493 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4c7171d-f08d-484e-8201-2e8644360ede-istio-token\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-r6v57\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:24.591372 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.591352 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz2bq\" (UniqueName: \"kubernetes.io/projected/a4c7171d-f08d-484e-8201-2e8644360ede-kube-api-access-cz2bq\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-r6v57\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:24.662071 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.662041 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:24.697723 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.697691 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:24.995553 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:41:24.995524 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4c7171d_f08d_484e_8201_2e8644360ede.slice/crio-d5fa11ab8dc6802543208ceab91c943ea22ac36275dc93d77bc71c2b04c0e055 WatchSource:0}: Error finding container d5fa11ab8dc6802543208ceab91c943ea22ac36275dc93d77bc71c2b04c0e055: Status 404 returned error can't find the container with id d5fa11ab8dc6802543208ceab91c943ea22ac36275dc93d77bc71c2b04c0e055 Oct 30 17:41:24.996253 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.996226 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57"] Oct 30 17:41:24.998979 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.998943 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:41:24.999056 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.999011 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:41:24.999056 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:24.999044 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:41:25.037915 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:25.037886 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp"] Oct 30 17:41:25.047407 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:41:25.047379 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod123ee2af_0589_49d5_b1a5_3f68b6ac9432.slice/crio-ed8dd87a69caca6ac1b25c06560291306b3411951b9dc01d86d954dea9f290ba WatchSource:0}: Error finding container ed8dd87a69caca6ac1b25c06560291306b3411951b9dc01d86d954dea9f290ba: Status 404 returned error can't find the container with id ed8dd87a69caca6ac1b25c06560291306b3411951b9dc01d86d954dea9f290ba Oct 30 17:41:25.051067 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:25.051027 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:41:25.051167 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:25.051115 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:41:25.051167 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:25.051159 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:41:25.296225 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:25.296177 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" event={"ID":"123ee2af-0589-49d5-b1a5-3f68b6ac9432","Type":"ContainerStarted","Data":"ed8dd87a69caca6ac1b25c06560291306b3411951b9dc01d86d954dea9f290ba"} Oct 30 17:41:25.301118 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:25.301081 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" event={"ID":"a4c7171d-f08d-484e-8201-2e8644360ede","Type":"ContainerStarted","Data":"e756f1e9a4b253bbb0708699975e12b59bd6d0b9a4528997ca8aa8a4f64595cc"} Oct 30 17:41:25.301118 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:25.301147 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" event={"ID":"a4c7171d-f08d-484e-8201-2e8644360ede","Type":"ContainerStarted","Data":"d5fa11ab8dc6802543208ceab91c943ea22ac36275dc93d77bc71c2b04c0e055"} Oct 30 17:41:25.354295 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:25.354179 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" podStartSLOduration=1.354159916 podStartE2EDuration="1.354159916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:41:25.34890588 +0000 UTC m=+1743.401005811" watchObservedRunningTime="2026-10-30 17:41:25.354159916 +0000 UTC m=+1743.406259852" Oct 30 17:41:25.663240 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:25.663204 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:26.110879 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:26.110831 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7715de9f-65c1-44fb-b83a-692ec07b30a8" path="/var/lib/kubelet/pods/7715de9f-65c1-44fb-b83a-692ec07b30a8/volumes" Oct 30 17:41:26.439053 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:26.438970 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" event={"ID":"123ee2af-0589-49d5-b1a5-3f68b6ac9432","Type":"ContainerStarted","Data":"adcc89c84400eb9efb88e66ff2de5735f5bcb11be75b46714d6f96981422c326"} Oct 30 17:41:26.511723 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:26.511659 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" podStartSLOduration=2.5116364129999997 podStartE2EDuration="2.511636413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:41:26.503324928 +0000 UTC m=+1744.555424861" watchObservedRunningTime="2026-10-30 17:41:26.511636413 +0000 UTC m=+1744.563736348" Oct 30 17:41:26.669045 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:26.669015 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:26.698555 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:26.698476 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:26.704146 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:26.704118 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:27.451732 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:27.448331 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:27.451732 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:27.448991 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:27.451732 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:27.450026 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:27.456873 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:27.453856 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:31.489117 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:31.489074 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-74762d-dd9c689f5-2wvwk"] Oct 30 17:41:31.495647 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:31.495622 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-74762d-dd9c689f5-2wvwk" Oct 30 17:41:31.514788 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:31.514753 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-74762d-dd9c689f5-2wvwk"] Oct 30 17:41:31.518473 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:31.518451 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-74762d\"" Oct 30 17:41:31.518473 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:31.518451 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-74762d-dockercfg-6h7s6\"" Oct 30 17:41:31.733823 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:31.731010 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fx89\" (UniqueName: \"kubernetes.io/projected/931d5477-513c-41d3-93ec-e1f24b4ff8f4-kube-api-access-7fx89\") pod \"payload-pre-processing-e2e-api-b-74762d-dd9c689f5-2wvwk\" (UID: \"931d5477-513c-41d3-93ec-e1f24b4ff8f4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-74762d-dd9c689f5-2wvwk" Oct 30 17:41:31.733823 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:31.731351 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/931d5477-513c-41d3-93ec-e1f24b4ff8f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-74762d-dd9c689f5-2wvwk\" (UID: \"931d5477-513c-41d3-93ec-e1f24b4ff8f4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-74762d-dd9c689f5-2wvwk" Oct 30 17:41:31.847324 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:31.846955 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/931d5477-513c-41d3-93ec-e1f24b4ff8f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-74762d-dd9c689f5-2wvwk\" (UID: \"931d5477-513c-41d3-93ec-e1f24b4ff8f4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-74762d-dd9c689f5-2wvwk" Oct 30 17:41:31.847324 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:31.847277 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fx89\" (UniqueName: \"kubernetes.io/projected/931d5477-513c-41d3-93ec-e1f24b4ff8f4-kube-api-access-7fx89\") pod \"payload-pre-processing-e2e-api-b-74762d-dd9c689f5-2wvwk\" (UID: \"931d5477-513c-41d3-93ec-e1f24b4ff8f4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-74762d-dd9c689f5-2wvwk" Oct 30 17:41:31.861884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:31.848810 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/931d5477-513c-41d3-93ec-e1f24b4ff8f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-74762d-dd9c689f5-2wvwk\" (UID: \"931d5477-513c-41d3-93ec-e1f24b4ff8f4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-74762d-dd9c689f5-2wvwk" Oct 30 17:41:31.915702 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:31.915668 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fx89\" (UniqueName: \"kubernetes.io/projected/931d5477-513c-41d3-93ec-e1f24b4ff8f4-kube-api-access-7fx89\") pod \"payload-pre-processing-e2e-api-b-74762d-dd9c689f5-2wvwk\" (UID: \"931d5477-513c-41d3-93ec-e1f24b4ff8f4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-74762d-dd9c689f5-2wvwk" Oct 30 17:41:32.118434 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:32.118362 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-74762d-dd9c689f5-2wvwk" Oct 30 17:41:32.445587 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:32.445556 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-74762d-dd9c689f5-2wvwk"] Oct 30 17:41:32.445711 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:41:32.445679 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod931d5477_513c_41d3_93ec_e1f24b4ff8f4.slice/crio-5a460e38627c9fb73070ffc2a204bbd19e3c6d987fa888e540af75f85a340373 WatchSource:0}: Error finding container 5a460e38627c9fb73070ffc2a204bbd19e3c6d987fa888e540af75f85a340373: Status 404 returned error can't find the container with id 5a460e38627c9fb73070ffc2a204bbd19e3c6d987fa888e540af75f85a340373 Oct 30 17:41:32.490033 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:32.490002 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-74762d-dd9c689f5-2wvwk" event={"ID":"931d5477-513c-41d3-93ec-e1f24b4ff8f4","Type":"ContainerStarted","Data":"5a460e38627c9fb73070ffc2a204bbd19e3c6d987fa888e540af75f85a340373"} Oct 30 17:41:33.499273 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:33.499184 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-74762d-dd9c689f5-2wvwk" event={"ID":"931d5477-513c-41d3-93ec-e1f24b4ff8f4","Type":"ContainerStarted","Data":"18b4fb3a3db654e5a06afed2c1c68a984c3c71953e4625ac21c33a23cfe53140"} Oct 30 17:41:33.499664 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:33.499276 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-74762d-dd9c689f5-2wvwk" Oct 30 17:41:33.541762 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:33.541713 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-74762d-dd9c689f5-2wvwk" podStartSLOduration=2.541699772 podStartE2EDuration="2.541699772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:41:33.53957937 +0000 UTC m=+1751.591679300" watchObservedRunningTime="2026-10-30 17:41:33.541699772 +0000 UTC m=+1751.593799703" Oct 30 17:41:35.997619 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:41:35.997580 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:41:44.510694 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:44.510663 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-74762d-dd9c689f5-2wvwk" Oct 30 17:41:48.770591 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:48.770546 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c9bdd9-5748f769dd-wl8td"] Oct 30 17:41:48.785606 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:48.785562 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c9bdd9-56b7d67c7f-r2hzh"] Oct 30 17:41:53.771397 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:53.771348 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c9bdd9-5748f769dd-wl8td" podUID="174ed90c-b452-485c-9dfb-af6a167089b0" containerName="payload-processing" containerID="cri-o://634e281bf586356d2ee7b469b770959c2542d90ffaf0f61fa49b6f894c2c75d9" gracePeriod=25 Oct 30 17:41:53.786907 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:53.786848 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9bdd9-56b7d67c7f-r2hzh" podUID="9960af67-a570-4962-a0ac-770879a98949" containerName="payload-pre-processing" containerID="cri-o://2bcfb1fb6157b56a7dbda3713b6930f8dc925de8ea50639d593f78e9c13150cb" gracePeriod=25 Oct 30 17:41:53.979815 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:53.979768 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57"] Oct 30 17:41:53.980204 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:53.980058 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" podUID="a4c7171d-f08d-484e-8201-2e8644360ede" containerName="istio-proxy" containerID="cri-o://e756f1e9a4b253bbb0708699975e12b59bd6d0b9a4528997ca8aa8a4f64595cc" gracePeriod=30 Oct 30 17:41:54.032969 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.032206 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f"] Oct 30 17:41:54.032969 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.032380 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp"] Oct 30 17:41:54.032969 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.032594 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" podUID="123ee2af-0589-49d5-b1a5-3f68b6ac9432" containerName="istio-proxy" containerID="cri-o://adcc89c84400eb9efb88e66ff2de5735f5bcb11be75b46714d6f96981422c326" gracePeriod=30 Oct 30 17:41:54.032969 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.032707 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" podUID="8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d" containerName="istio-proxy" containerID="cri-o://f4e4b8cd86263b1c642bdd347d9e0ffd208835c744a40828bf221b1da7ab1546" gracePeriod=30 Oct 30 17:41:54.041304 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.039788 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7"] Oct 30 17:41:54.041304 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.040058 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" podUID="09efdc53-164b-4276-8179-5dffd1c33896" containerName="istio-proxy" containerID="cri-o://7d7b88394f872a7206a1d498246fda0b5a7e3db3b6b088b86147cc1d3fb76328" gracePeriod=30 Oct 30 17:41:54.576980 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.576946 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c9bdd9-5748f769dd-wl8td" Oct 30 17:41:54.646169 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.646142 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9bdd9-56b7d67c7f-r2hzh" Oct 30 17:41:54.728264 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.728188 4965 generic.go:356] "Generic (PLEG): container finished" podID="174ed90c-b452-485c-9dfb-af6a167089b0" containerID="634e281bf586356d2ee7b469b770959c2542d90ffaf0f61fa49b6f894c2c75d9" exitCode=0 Oct 30 17:41:54.728264 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.728233 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c9bdd9-5748f769dd-wl8td" Oct 30 17:41:54.728493 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.728265 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c9bdd9-5748f769dd-wl8td" event={"ID":"174ed90c-b452-485c-9dfb-af6a167089b0","Type":"ContainerDied","Data":"634e281bf586356d2ee7b469b770959c2542d90ffaf0f61fa49b6f894c2c75d9"} Oct 30 17:41:54.728493 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.728319 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c9bdd9-5748f769dd-wl8td" event={"ID":"174ed90c-b452-485c-9dfb-af6a167089b0","Type":"ContainerDied","Data":"a424fce8dc925678f8891ad7cfc58e26fa2f17d606dda0d23c7ddba274c96f2b"} Oct 30 17:41:54.728493 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.728348 4965 scope.go:122] "RemoveContainer" containerID="634e281bf586356d2ee7b469b770959c2542d90ffaf0f61fa49b6f894c2c75d9" Oct 30 17:41:54.729723 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.729700 4965 generic.go:356] "Generic (PLEG): container finished" podID="9960af67-a570-4962-a0ac-770879a98949" containerID="2bcfb1fb6157b56a7dbda3713b6930f8dc925de8ea50639d593f78e9c13150cb" exitCode=0 Oct 30 17:41:54.729804 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.729745 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9bdd9-56b7d67c7f-r2hzh" event={"ID":"9960af67-a570-4962-a0ac-770879a98949","Type":"ContainerDied","Data":"2bcfb1fb6157b56a7dbda3713b6930f8dc925de8ea50639d593f78e9c13150cb"} Oct 30 17:41:54.729804 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.729767 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9bdd9-56b7d67c7f-r2hzh" Oct 30 17:41:54.729804 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.729796 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9bdd9-56b7d67c7f-r2hzh" event={"ID":"9960af67-a570-4962-a0ac-770879a98949","Type":"ContainerDied","Data":"dc426b12e8990109e9a810efc10d1f6e002c28afd940fe1901d2b38b98804619"} Oct 30 17:41:54.744388 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.744296 4965 scope.go:122] "RemoveContainer" containerID="634e281bf586356d2ee7b469b770959c2542d90ffaf0f61fa49b6f894c2c75d9" Oct 30 17:41:54.744741 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:41:54.744719 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"634e281bf586356d2ee7b469b770959c2542d90ffaf0f61fa49b6f894c2c75d9\": container with ID starting with 634e281bf586356d2ee7b469b770959c2542d90ffaf0f61fa49b6f894c2c75d9 not found: ID does not exist" containerID="634e281bf586356d2ee7b469b770959c2542d90ffaf0f61fa49b6f894c2c75d9" Oct 30 17:41:54.744847 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.744747 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"634e281bf586356d2ee7b469b770959c2542d90ffaf0f61fa49b6f894c2c75d9"} err="failed to get container status \"634e281bf586356d2ee7b469b770959c2542d90ffaf0f61fa49b6f894c2c75d9\": rpc error: code = NotFound desc = could not find container \"634e281bf586356d2ee7b469b770959c2542d90ffaf0f61fa49b6f894c2c75d9\": container with ID starting with 634e281bf586356d2ee7b469b770959c2542d90ffaf0f61fa49b6f894c2c75d9 not found: ID does not exist" Oct 30 17:41:54.744847 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.744766 4965 scope.go:122] "RemoveContainer" containerID="2bcfb1fb6157b56a7dbda3713b6930f8dc925de8ea50639d593f78e9c13150cb" Oct 30 17:41:54.755363 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.755328 4965 scope.go:122] "RemoveContainer" containerID="2bcfb1fb6157b56a7dbda3713b6930f8dc925de8ea50639d593f78e9c13150cb" Oct 30 17:41:54.755647 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:41:54.755627 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bcfb1fb6157b56a7dbda3713b6930f8dc925de8ea50639d593f78e9c13150cb\": container with ID starting with 2bcfb1fb6157b56a7dbda3713b6930f8dc925de8ea50639d593f78e9c13150cb not found: ID does not exist" containerID="2bcfb1fb6157b56a7dbda3713b6930f8dc925de8ea50639d593f78e9c13150cb" Oct 30 17:41:54.755696 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.755659 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bcfb1fb6157b56a7dbda3713b6930f8dc925de8ea50639d593f78e9c13150cb"} err="failed to get container status \"2bcfb1fb6157b56a7dbda3713b6930f8dc925de8ea50639d593f78e9c13150cb\": rpc error: code = NotFound desc = could not find container \"2bcfb1fb6157b56a7dbda3713b6930f8dc925de8ea50639d593f78e9c13150cb\": container with ID starting with 2bcfb1fb6157b56a7dbda3713b6930f8dc925de8ea50639d593f78e9c13150cb not found: ID does not exist" Oct 30 17:41:54.772929 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.772899 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9960af67-a570-4962-a0ac-770879a98949-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9960af67-a570-4962-a0ac-770879a98949-plugins-config-volume\") pod \"9960af67-a570-4962-a0ac-770879a98949\" (UID: \"9960af67-a570-4962-a0ac-770879a98949\") " Oct 30 17:41:54.773327 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.772959 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/174ed90c-b452-485c-9dfb-af6a167089b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/174ed90c-b452-485c-9dfb-af6a167089b0-plugins-config-volume\") pod \"174ed90c-b452-485c-9dfb-af6a167089b0\" (UID: \"174ed90c-b452-485c-9dfb-af6a167089b0\") " Oct 30 17:41:54.773327 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.772995 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/174ed90c-b452-485c-9dfb-af6a167089b0-kube-api-access-8jmvw\" (UniqueName: \"kubernetes.io/projected/174ed90c-b452-485c-9dfb-af6a167089b0-kube-api-access-8jmvw\") pod \"174ed90c-b452-485c-9dfb-af6a167089b0\" (UID: \"174ed90c-b452-485c-9dfb-af6a167089b0\") " Oct 30 17:41:54.773327 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.773144 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9960af67-a570-4962-a0ac-770879a98949-kube-api-access-dlbdl\" (UniqueName: \"kubernetes.io/projected/9960af67-a570-4962-a0ac-770879a98949-kube-api-access-dlbdl\") pod \"9960af67-a570-4962-a0ac-770879a98949\" (UID: \"9960af67-a570-4962-a0ac-770879a98949\") " Oct 30 17:41:54.773327 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.773258 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9960af67-a570-4962-a0ac-770879a98949-plugins-config-volume" pod "9960af67-a570-4962-a0ac-770879a98949" (UID: "9960af67-a570-4962-a0ac-770879a98949"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:41:54.773509 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.773348 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/174ed90c-b452-485c-9dfb-af6a167089b0-plugins-config-volume" pod "174ed90c-b452-485c-9dfb-af6a167089b0" (UID: "174ed90c-b452-485c-9dfb-af6a167089b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:41:54.773829 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.773812 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9960af67-a570-4962-a0ac-770879a98949-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:54.773904 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.773834 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/174ed90c-b452-485c-9dfb-af6a167089b0-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:54.775204 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.775182 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9960af67-a570-4962-a0ac-770879a98949-kube-api-access-dlbdl" pod "9960af67-a570-4962-a0ac-770879a98949" (UID: "9960af67-a570-4962-a0ac-770879a98949"). InnerVolumeSpecName "kube-api-access-dlbdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:54.775287 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.775251 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/174ed90c-b452-485c-9dfb-af6a167089b0-kube-api-access-8jmvw" pod "174ed90c-b452-485c-9dfb-af6a167089b0" (UID: "174ed90c-b452-485c-9dfb-af6a167089b0"). InnerVolumeSpecName "kube-api-access-8jmvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:54.875223 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.875182 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jmvw\" (UniqueName: \"kubernetes.io/projected/174ed90c-b452-485c-9dfb-af6a167089b0-kube-api-access-8jmvw\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:54.875223 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:54.875220 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlbdl\" (UniqueName: \"kubernetes.io/projected/9960af67-a570-4962-a0ac-770879a98949-kube-api-access-dlbdl\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:55.080542 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:55.080499 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c9bdd9-5748f769dd-wl8td"] Oct 30 17:41:55.085327 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:55.085304 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c9bdd9-5748f769dd-wl8td"] Oct 30 17:41:55.124174 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:55.124140 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c9bdd9-56b7d67c7f-r2hzh"] Oct 30 17:41:55.130415 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:55.130385 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c9bdd9-56b7d67c7f-r2hzh"] Oct 30 17:41:56.101044 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:56.101013 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="174ed90c-b452-485c-9dfb-af6a167089b0" path="/var/lib/kubelet/pods/174ed90c-b452-485c-9dfb-af6a167089b0/volumes" Oct 30 17:41:56.101529 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:56.101511 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9960af67-a570-4962-a0ac-770879a98949" path="/var/lib/kubelet/pods/9960af67-a570-4962-a0ac-770879a98949/volumes" Oct 30 17:41:57.450686 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:57.450648 4965 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:41:57.451075 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:57.450708 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" podUID="123ee2af-0589-49d5-b1a5-3f68b6ac9432" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:41:57.451075 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:57.450899 4965 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Oct 30 17:41:57.451075 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:57.450948 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" podUID="a4c7171d-f08d-484e-8201-2e8644360ede" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:41:59.401132 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.401098 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:41:59.405062 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.405033 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:59.486411 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.486383 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:59.490402 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.490371 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istio-envoy\") pod \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " Oct 30 17:41:59.490546 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.490426 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-workload-certs\") pod \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " Oct 30 17:41:59.490546 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.490481 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/123ee2af-0589-49d5-b1a5-3f68b6ac9432-kube-api-access-49k48\" (UniqueName: \"kubernetes.io/projected/123ee2af-0589-49d5-b1a5-3f68b6ac9432-kube-api-access-49k48\") pod \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " Oct 30 17:41:59.490546 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.490541 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-kube-api-access-sk7lz\" (UniqueName: \"kubernetes.io/projected/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-kube-api-access-sk7lz\") pod \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " Oct 30 17:41:59.490734 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.490630 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-workload-certs\") pod \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " Oct 30 17:41:59.490734 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.490668 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-credential-socket\") pod \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " Oct 30 17:41:59.490734 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.490699 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istio-envoy\") pod \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " Oct 30 17:41:59.490734 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.490726 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istio-token\" (UniqueName: \"kubernetes.io/projected/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istio-token\") pod \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " Oct 30 17:41:59.491065 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.490756 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istio-data\") pod \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " Oct 30 17:41:59.491065 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.490832 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istiod-ca-cert\") pod \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " Oct 30 17:41:59.491065 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.490973 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istio-podinfo\") pod \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " Oct 30 17:41:59.491065 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.491031 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-credential-socket\") pod \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " Oct 30 17:41:59.491065 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.491050 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-workload-certs" pod "123ee2af-0589-49d5-b1a5-3f68b6ac9432" (UID: "123ee2af-0589-49d5-b1a5-3f68b6ac9432"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.491336 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.491069 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istio-data\") pod \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " Oct 30 17:41:59.491336 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.491104 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istiod-ca-cert\") pod \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " Oct 30 17:41:59.491336 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.491106 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:59.491336 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.491156 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istio-token\" (UniqueName: \"kubernetes.io/projected/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istio-token\") pod \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " Oct 30 17:41:59.491336 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.491184 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istio-podinfo\") pod \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " Oct 30 17:41:59.491336 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.491243 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-workload-socket\") pod \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\" (UID: \"123ee2af-0589-49d5-b1a5-3f68b6ac9432\") " Oct 30 17:41:59.491336 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.491302 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-workload-socket\") pod \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\" (UID: \"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d\") " Oct 30 17:41:59.492180 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.492015 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-credential-socket" pod "8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d" (UID: "8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.492280 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.492200 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.492280 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.492222 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.492539 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.492410 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-workload-socket" pod "123ee2af-0589-49d5-b1a5-3f68b6ac9432" (UID: "123ee2af-0589-49d5-b1a5-3f68b6ac9432"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.492539 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.492476 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istio-data" pod "8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d" (UID: "8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.492680 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.492583 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-workload-certs" pod "8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d" (UID: "8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.492680 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.492650 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-workload-socket" pod "8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d" (UID: "8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.492875 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.492793 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-credential-socket" pod "123ee2af-0589-49d5-b1a5-3f68b6ac9432" (UID: "123ee2af-0589-49d5-b1a5-3f68b6ac9432"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.492875 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.492803 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istio-data" pod "123ee2af-0589-49d5-b1a5-3f68b6ac9432" (UID: "123ee2af-0589-49d5-b1a5-3f68b6ac9432"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.492875 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.492836 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istiod-ca-cert" pod "8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d" (UID: "8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:41:59.493162 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.493117 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istiod-ca-cert" pod "123ee2af-0589-49d5-b1a5-3f68b6ac9432" (UID: "123ee2af-0589-49d5-b1a5-3f68b6ac9432"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:41:59.495242 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.495215 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istio-envoy" pod "8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d" (UID: "8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.497004 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.496981 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istio-podinfo" pod "123ee2af-0589-49d5-b1a5-3f68b6ac9432" (UID: "123ee2af-0589-49d5-b1a5-3f68b6ac9432"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:41:59.497190 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.497163 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istio-envoy" pod "123ee2af-0589-49d5-b1a5-3f68b6ac9432" (UID: "123ee2af-0589-49d5-b1a5-3f68b6ac9432"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.497270 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.497186 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/123ee2af-0589-49d5-b1a5-3f68b6ac9432-kube-api-access-49k48" pod "123ee2af-0589-49d5-b1a5-3f68b6ac9432" (UID: "123ee2af-0589-49d5-b1a5-3f68b6ac9432"). InnerVolumeSpecName "kube-api-access-49k48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:59.497270 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.497229 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istio-token" pod "123ee2af-0589-49d5-b1a5-3f68b6ac9432" (UID: "123ee2af-0589-49d5-b1a5-3f68b6ac9432"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:59.497405 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.497352 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-kube-api-access-sk7lz" pod "8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d" (UID: "8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d"). InnerVolumeSpecName "kube-api-access-sk7lz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:59.497542 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.497519 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istio-token" pod "8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d" (UID: "8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:59.498607 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.498577 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istio-podinfo" pod "8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d" (UID: "8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:41:59.593790 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.593754 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-workload-certs\") pod \"a4c7171d-f08d-484e-8201-2e8644360ede\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " Oct 30 17:41:59.594020 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.593807 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4c7171d-f08d-484e-8201-2e8644360ede-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4c7171d-f08d-484e-8201-2e8644360ede-istio-podinfo\") pod \"a4c7171d-f08d-484e-8201-2e8644360ede\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " Oct 30 17:41:59.594020 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.593890 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-istio-data\") pod \"09efdc53-164b-4276-8179-5dffd1c33896\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " Oct 30 17:41:59.594020 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.593952 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4c7171d-f08d-484e-8201-2e8644360ede-kube-api-access-cz2bq\" (UniqueName: \"kubernetes.io/projected/a4c7171d-f08d-484e-8201-2e8644360ede-kube-api-access-cz2bq\") pod \"a4c7171d-f08d-484e-8201-2e8644360ede\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " Oct 30 17:41:59.594020 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.593986 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09efdc53-164b-4276-8179-5dffd1c33896-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09efdc53-164b-4276-8179-5dffd1c33896-istio-podinfo\") pod \"09efdc53-164b-4276-8179-5dffd1c33896\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " Oct 30 17:41:59.594281 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.594031 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09efdc53-164b-4276-8179-5dffd1c33896-kube-api-access-qhdm5\" (UniqueName: \"kubernetes.io/projected/09efdc53-164b-4276-8179-5dffd1c33896-kube-api-access-qhdm5\") pod \"09efdc53-164b-4276-8179-5dffd1c33896\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " Oct 30 17:41:59.594281 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.594087 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-workload-socket\") pod \"09efdc53-164b-4276-8179-5dffd1c33896\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " Oct 30 17:41:59.594281 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.594166 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4c7171d-f08d-484e-8201-2e8644360ede-istio-token\" (UniqueName: \"kubernetes.io/projected/a4c7171d-f08d-484e-8201-2e8644360ede-istio-token\") pod \"a4c7171d-f08d-484e-8201-2e8644360ede\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " Oct 30 17:41:59.594281 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.594221 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-credential-socket\") pod \"09efdc53-164b-4276-8179-5dffd1c33896\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " Oct 30 17:41:59.594281 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.594261 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-workload-certs\") pod \"09efdc53-164b-4276-8179-5dffd1c33896\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " Oct 30 17:41:59.594535 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.594292 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09efdc53-164b-4276-8179-5dffd1c33896-istio-token\" (UniqueName: \"kubernetes.io/projected/09efdc53-164b-4276-8179-5dffd1c33896-istio-token\") pod \"09efdc53-164b-4276-8179-5dffd1c33896\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " Oct 30 17:41:59.594535 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.594353 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-istio-data\") pod \"a4c7171d-f08d-484e-8201-2e8644360ede\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " Oct 30 17:41:59.594535 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.594387 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-credential-socket\") pod \"a4c7171d-f08d-484e-8201-2e8644360ede\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " Oct 30 17:41:59.594535 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.594426 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09efdc53-164b-4276-8179-5dffd1c33896-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09efdc53-164b-4276-8179-5dffd1c33896-istiod-ca-cert\") pod \"09efdc53-164b-4276-8179-5dffd1c33896\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " Oct 30 17:41:59.594749 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.594546 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4c7171d-f08d-484e-8201-2e8644360ede-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4c7171d-f08d-484e-8201-2e8644360ede-istiod-ca-cert\") pod \"a4c7171d-f08d-484e-8201-2e8644360ede\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " Oct 30 17:41:59.594749 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.594606 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-istio-envoy\") pod \"09efdc53-164b-4276-8179-5dffd1c33896\" (UID: \"09efdc53-164b-4276-8179-5dffd1c33896\") " Oct 30 17:41:59.594749 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.594636 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-workload-socket\") pod \"a4c7171d-f08d-484e-8201-2e8644360ede\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " Oct 30 17:41:59.594932 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.594779 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-istio-envoy\") pod \"a4c7171d-f08d-484e-8201-2e8644360ede\" (UID: \"a4c7171d-f08d-484e-8201-2e8644360ede\") " Oct 30 17:41:59.595726 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.594083 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-workload-certs" pod "a4c7171d-f08d-484e-8201-2e8644360ede" (UID: "a4c7171d-f08d-484e-8201-2e8644360ede"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.595726 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.594465 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-istio-data" pod "09efdc53-164b-4276-8179-5dffd1c33896" (UID: "09efdc53-164b-4276-8179-5dffd1c33896"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.595726 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.594998 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-istio-data" pod "a4c7171d-f08d-484e-8201-2e8644360ede" (UID: "a4c7171d-f08d-484e-8201-2e8644360ede"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.595726 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.595306 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-credential-socket" pod "09efdc53-164b-4276-8179-5dffd1c33896" (UID: "09efdc53-164b-4276-8179-5dffd1c33896"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.595726 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.595405 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4c7171d-f08d-484e-8201-2e8644360ede-istiod-ca-cert" pod "a4c7171d-f08d-484e-8201-2e8644360ede" (UID: "a4c7171d-f08d-484e-8201-2e8644360ede"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:41:59.595726 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.595503 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-workload-socket" pod "a4c7171d-f08d-484e-8201-2e8644360ede" (UID: "a4c7171d-f08d-484e-8201-2e8644360ede"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.595726 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.595524 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-credential-socket" pod "a4c7171d-f08d-484e-8201-2e8644360ede" (UID: "a4c7171d-f08d-484e-8201-2e8644360ede"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.595726 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.595674 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-workload-socket" pod "09efdc53-164b-4276-8179-5dffd1c33896" (UID: "09efdc53-164b-4276-8179-5dffd1c33896"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.596312 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.596091 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09efdc53-164b-4276-8179-5dffd1c33896-istiod-ca-cert" pod "09efdc53-164b-4276-8179-5dffd1c33896" (UID: "09efdc53-164b-4276-8179-5dffd1c33896"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:41:59.596312 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.596143 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-workload-certs" pod "09efdc53-164b-4276-8179-5dffd1c33896" (UID: "09efdc53-164b-4276-8179-5dffd1c33896"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.597034 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.597007 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.597135 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.597039 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.597135 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.597053 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.597135 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.597068 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49k48\" (UniqueName: \"kubernetes.io/projected/123ee2af-0589-49d5-b1a5-3f68b6ac9432-kube-api-access-49k48\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.597135 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.597081 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.597135 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.597101 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09efdc53-164b-4276-8179-5dffd1c33896-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.597135 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.597114 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk7lz\" (UniqueName: \"kubernetes.io/projected/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-kube-api-access-sk7lz\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.597135 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.597129 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4c7171d-f08d-484e-8201-2e8644360ede-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.597572 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.597141 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.597572 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.597178 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.597572 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.597193 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.597572 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.597206 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.597572 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.597219 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.597572 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.597231 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.597572 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.597243 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.597572 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.597257 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.597572 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.597268 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.597572 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.597280 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.597572 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.597292 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.597572 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.597304 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.597572 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.597317 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.597572 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.597331 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.597572 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.597344 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.597572 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.597357 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.597572 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.597371 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.597572 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.597384 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/123ee2af-0589-49d5-b1a5-3f68b6ac9432-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.597572 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.597449 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4c7171d-f08d-484e-8201-2e8644360ede-istio-podinfo" pod "a4c7171d-f08d-484e-8201-2e8644360ede" (UID: "a4c7171d-f08d-484e-8201-2e8644360ede"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:41:59.598772 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.597596 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4c7171d-f08d-484e-8201-2e8644360ede-kube-api-access-cz2bq" pod "a4c7171d-f08d-484e-8201-2e8644360ede" (UID: "a4c7171d-f08d-484e-8201-2e8644360ede"). InnerVolumeSpecName "kube-api-access-cz2bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:59.598772 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.598017 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4c7171d-f08d-484e-8201-2e8644360ede-istio-token" pod "a4c7171d-f08d-484e-8201-2e8644360ede" (UID: "a4c7171d-f08d-484e-8201-2e8644360ede"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:59.599678 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.599653 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09efdc53-164b-4276-8179-5dffd1c33896-kube-api-access-qhdm5" pod "09efdc53-164b-4276-8179-5dffd1c33896" (UID: "09efdc53-164b-4276-8179-5dffd1c33896"). InnerVolumeSpecName "kube-api-access-qhdm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:59.600009 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.599986 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-istio-envoy" pod "a4c7171d-f08d-484e-8201-2e8644360ede" (UID: "a4c7171d-f08d-484e-8201-2e8644360ede"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.601504 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.601478 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09efdc53-164b-4276-8179-5dffd1c33896-istio-token" pod "09efdc53-164b-4276-8179-5dffd1c33896" (UID: "09efdc53-164b-4276-8179-5dffd1c33896"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:59.602138 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.602118 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09efdc53-164b-4276-8179-5dffd1c33896-istio-podinfo" pod "09efdc53-164b-4276-8179-5dffd1c33896" (UID: "09efdc53-164b-4276-8179-5dffd1c33896"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:41:59.603140 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.603117 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-istio-envoy" pod "09efdc53-164b-4276-8179-5dffd1c33896" (UID: "09efdc53-164b-4276-8179-5dffd1c33896"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.699061 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.699030 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09efdc53-164b-4276-8179-5dffd1c33896-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.699061 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.699056 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4c7171d-f08d-484e-8201-2e8644360ede-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.699061 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.699065 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4c7171d-f08d-484e-8201-2e8644360ede-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.699611 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.699074 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz2bq\" (UniqueName: \"kubernetes.io/projected/a4c7171d-f08d-484e-8201-2e8644360ede-kube-api-access-cz2bq\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.699611 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.699083 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09efdc53-164b-4276-8179-5dffd1c33896-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.699611 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.699091 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhdm5\" (UniqueName: \"kubernetes.io/projected/09efdc53-164b-4276-8179-5dffd1c33896-kube-api-access-qhdm5\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.699611 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.699099 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4c7171d-f08d-484e-8201-2e8644360ede-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.699611 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.699107 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09efdc53-164b-4276-8179-5dffd1c33896-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.772061 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.771984 4965 generic.go:356] "Generic (PLEG): container finished" podID="123ee2af-0589-49d5-b1a5-3f68b6ac9432" containerID="adcc89c84400eb9efb88e66ff2de5735f5bcb11be75b46714d6f96981422c326" exitCode=0 Oct 30 17:41:59.772061 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.772031 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" Oct 30 17:41:59.772281 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.772068 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" event={"ID":"123ee2af-0589-49d5-b1a5-3f68b6ac9432","Type":"ContainerDied","Data":"adcc89c84400eb9efb88e66ff2de5735f5bcb11be75b46714d6f96981422c326"} Oct 30 17:41:59.772281 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.772123 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp" event={"ID":"123ee2af-0589-49d5-b1a5-3f68b6ac9432","Type":"ContainerDied","Data":"ed8dd87a69caca6ac1b25c06560291306b3411951b9dc01d86d954dea9f290ba"} Oct 30 17:41:59.772281 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.772155 4965 scope.go:122] "RemoveContainer" containerID="adcc89c84400eb9efb88e66ff2de5735f5bcb11be75b46714d6f96981422c326" Oct 30 17:41:59.773621 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.773602 4965 generic.go:356] "Generic (PLEG): container finished" podID="a4c7171d-f08d-484e-8201-2e8644360ede" containerID="e756f1e9a4b253bbb0708699975e12b59bd6d0b9a4528997ca8aa8a4f64595cc" exitCode=0 Oct 30 17:41:59.773705 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.773680 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" Oct 30 17:41:59.773705 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.773687 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" event={"ID":"a4c7171d-f08d-484e-8201-2e8644360ede","Type":"ContainerDied","Data":"e756f1e9a4b253bbb0708699975e12b59bd6d0b9a4528997ca8aa8a4f64595cc"} Oct 30 17:41:59.773798 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.773731 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57" event={"ID":"a4c7171d-f08d-484e-8201-2e8644360ede","Type":"ContainerDied","Data":"d5fa11ab8dc6802543208ceab91c943ea22ac36275dc93d77bc71c2b04c0e055"} Oct 30 17:41:59.777502 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.777480 4965 generic.go:356] "Generic (PLEG): container finished" podID="8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d" containerID="f4e4b8cd86263b1c642bdd347d9e0ffd208835c744a40828bf221b1da7ab1546" exitCode=0 Oct 30 17:41:59.777609 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.777536 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" Oct 30 17:41:59.777609 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.777575 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" event={"ID":"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d","Type":"ContainerDied","Data":"f4e4b8cd86263b1c642bdd347d9e0ffd208835c744a40828bf221b1da7ab1546"} Oct 30 17:41:59.777709 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.777621 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f" event={"ID":"8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d","Type":"ContainerDied","Data":"b35e45ce0319d283ebfd9b1214a7e24dbc7344e449e94637b20602ae0fcb1270"} Oct 30 17:41:59.780127 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.780101 4965 generic.go:356] "Generic (PLEG): container finished" podID="09efdc53-164b-4276-8179-5dffd1c33896" containerID="7d7b88394f872a7206a1d498246fda0b5a7e3db3b6b088b86147cc1d3fb76328" exitCode=0 Oct 30 17:41:59.780246 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.780134 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" event={"ID":"09efdc53-164b-4276-8179-5dffd1c33896","Type":"ContainerDied","Data":"7d7b88394f872a7206a1d498246fda0b5a7e3db3b6b088b86147cc1d3fb76328"} Oct 30 17:41:59.780246 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.780151 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" Oct 30 17:41:59.780246 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.780186 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7" event={"ID":"09efdc53-164b-4276-8179-5dffd1c33896","Type":"ContainerDied","Data":"2a13a346b0392533f804be12cdb95c7fffad438052b80ac7ecc157463c556e87"} Oct 30 17:41:59.785973 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.785950 4965 scope.go:122] "RemoveContainer" containerID="adcc89c84400eb9efb88e66ff2de5735f5bcb11be75b46714d6f96981422c326" Oct 30 17:41:59.786253 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:41:59.786226 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adcc89c84400eb9efb88e66ff2de5735f5bcb11be75b46714d6f96981422c326\": container with ID starting with adcc89c84400eb9efb88e66ff2de5735f5bcb11be75b46714d6f96981422c326 not found: ID does not exist" containerID="adcc89c84400eb9efb88e66ff2de5735f5bcb11be75b46714d6f96981422c326" Oct 30 17:41:59.786353 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.786261 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adcc89c84400eb9efb88e66ff2de5735f5bcb11be75b46714d6f96981422c326"} err="failed to get container status \"adcc89c84400eb9efb88e66ff2de5735f5bcb11be75b46714d6f96981422c326\": rpc error: code = NotFound desc = could not find container \"adcc89c84400eb9efb88e66ff2de5735f5bcb11be75b46714d6f96981422c326\": container with ID starting with adcc89c84400eb9efb88e66ff2de5735f5bcb11be75b46714d6f96981422c326 not found: ID does not exist" Oct 30 17:41:59.786353 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.786281 4965 scope.go:122] "RemoveContainer" containerID="e756f1e9a4b253bbb0708699975e12b59bd6d0b9a4528997ca8aa8a4f64595cc" Oct 30 17:41:59.797682 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.797665 4965 scope.go:122] "RemoveContainer" containerID="e756f1e9a4b253bbb0708699975e12b59bd6d0b9a4528997ca8aa8a4f64595cc" Oct 30 17:41:59.797976 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:41:59.797957 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e756f1e9a4b253bbb0708699975e12b59bd6d0b9a4528997ca8aa8a4f64595cc\": container with ID starting with e756f1e9a4b253bbb0708699975e12b59bd6d0b9a4528997ca8aa8a4f64595cc not found: ID does not exist" containerID="e756f1e9a4b253bbb0708699975e12b59bd6d0b9a4528997ca8aa8a4f64595cc" Oct 30 17:41:59.798034 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.797982 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e756f1e9a4b253bbb0708699975e12b59bd6d0b9a4528997ca8aa8a4f64595cc"} err="failed to get container status \"e756f1e9a4b253bbb0708699975e12b59bd6d0b9a4528997ca8aa8a4f64595cc\": rpc error: code = NotFound desc = could not find container \"e756f1e9a4b253bbb0708699975e12b59bd6d0b9a4528997ca8aa8a4f64595cc\": container with ID starting with e756f1e9a4b253bbb0708699975e12b59bd6d0b9a4528997ca8aa8a4f64595cc not found: ID does not exist" Oct 30 17:41:59.798034 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.797996 4965 scope.go:122] "RemoveContainer" containerID="f4e4b8cd86263b1c642bdd347d9e0ffd208835c744a40828bf221b1da7ab1546" Oct 30 17:41:59.807675 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.807657 4965 scope.go:122] "RemoveContainer" containerID="778ff37a1424c5358b1ceaa72ddecac6f3bb1906ce75db337be7471a380bc0d0" Oct 30 17:41:59.817082 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.817068 4965 scope.go:122] "RemoveContainer" containerID="f4e4b8cd86263b1c642bdd347d9e0ffd208835c744a40828bf221b1da7ab1546" Oct 30 17:41:59.817361 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:41:59.817342 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4e4b8cd86263b1c642bdd347d9e0ffd208835c744a40828bf221b1da7ab1546\": container with ID starting with f4e4b8cd86263b1c642bdd347d9e0ffd208835c744a40828bf221b1da7ab1546 not found: ID does not exist" containerID="f4e4b8cd86263b1c642bdd347d9e0ffd208835c744a40828bf221b1da7ab1546" Oct 30 17:41:59.817411 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.817372 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4e4b8cd86263b1c642bdd347d9e0ffd208835c744a40828bf221b1da7ab1546"} err="failed to get container status \"f4e4b8cd86263b1c642bdd347d9e0ffd208835c744a40828bf221b1da7ab1546\": rpc error: code = NotFound desc = could not find container \"f4e4b8cd86263b1c642bdd347d9e0ffd208835c744a40828bf221b1da7ab1546\": container with ID starting with f4e4b8cd86263b1c642bdd347d9e0ffd208835c744a40828bf221b1da7ab1546 not found: ID does not exist" Oct 30 17:41:59.817411 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.817388 4965 scope.go:122] "RemoveContainer" containerID="778ff37a1424c5358b1ceaa72ddecac6f3bb1906ce75db337be7471a380bc0d0" Oct 30 17:41:59.817631 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:41:59.817611 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"778ff37a1424c5358b1ceaa72ddecac6f3bb1906ce75db337be7471a380bc0d0\": container with ID starting with 778ff37a1424c5358b1ceaa72ddecac6f3bb1906ce75db337be7471a380bc0d0 not found: ID does not exist" containerID="778ff37a1424c5358b1ceaa72ddecac6f3bb1906ce75db337be7471a380bc0d0" Oct 30 17:41:59.817735 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.817635 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"778ff37a1424c5358b1ceaa72ddecac6f3bb1906ce75db337be7471a380bc0d0"} err="failed to get container status \"778ff37a1424c5358b1ceaa72ddecac6f3bb1906ce75db337be7471a380bc0d0\": rpc error: code = NotFound desc = could not find container \"778ff37a1424c5358b1ceaa72ddecac6f3bb1906ce75db337be7471a380bc0d0\": container with ID starting with 778ff37a1424c5358b1ceaa72ddecac6f3bb1906ce75db337be7471a380bc0d0 not found: ID does not exist" Oct 30 17:41:59.817735 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.817654 4965 scope.go:122] "RemoveContainer" containerID="7d7b88394f872a7206a1d498246fda0b5a7e3db3b6b088b86147cc1d3fb76328" Oct 30 17:41:59.827315 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.827297 4965 scope.go:122] "RemoveContainer" containerID="7d7b88394f872a7206a1d498246fda0b5a7e3db3b6b088b86147cc1d3fb76328" Oct 30 17:41:59.827556 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:41:59.827537 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d7b88394f872a7206a1d498246fda0b5a7e3db3b6b088b86147cc1d3fb76328\": container with ID starting with 7d7b88394f872a7206a1d498246fda0b5a7e3db3b6b088b86147cc1d3fb76328 not found: ID does not exist" containerID="7d7b88394f872a7206a1d498246fda0b5a7e3db3b6b088b86147cc1d3fb76328" Oct 30 17:41:59.827616 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.827563 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d7b88394f872a7206a1d498246fda0b5a7e3db3b6b088b86147cc1d3fb76328"} err="failed to get container status \"7d7b88394f872a7206a1d498246fda0b5a7e3db3b6b088b86147cc1d3fb76328\": rpc error: code = NotFound desc = could not find container \"7d7b88394f872a7206a1d498246fda0b5a7e3db3b6b088b86147cc1d3fb76328\": container with ID starting with 7d7b88394f872a7206a1d498246fda0b5a7e3db3b6b088b86147cc1d3fb76328 not found: ID does not exist" Oct 30 17:41:59.859657 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.859623 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp"] Oct 30 17:41:59.872254 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.872232 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-8zhzp"] Oct 30 17:41:59.925966 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.925935 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f"] Oct 30 17:41:59.930251 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.930217 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-sfq4f"] Oct 30 17:41:59.960181 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.960156 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57"] Oct 30 17:41:59.965490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:41:59.965468 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-r6v57"] Oct 30 17:42:00.130576 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:00.130541 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="123ee2af-0589-49d5-b1a5-3f68b6ac9432" path="/var/lib/kubelet/pods/123ee2af-0589-49d5-b1a5-3f68b6ac9432/volumes" Oct 30 17:42:00.131194 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:00.131171 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d" path="/var/lib/kubelet/pods/8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d/volumes" Oct 30 17:42:00.131671 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:00.131655 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4c7171d-f08d-484e-8201-2e8644360ede" path="/var/lib/kubelet/pods/a4c7171d-f08d-484e-8201-2e8644360ede/volumes" Oct 30 17:42:00.132363 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:00.132344 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7"] Oct 30 17:42:00.132445 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:00.132376 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-gxts7"] Oct 30 17:42:02.102197 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:02.102164 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09efdc53-164b-4276-8179-5dffd1c33896" path="/var/lib/kubelet/pods/09efdc53-164b-4276-8179-5dffd1c33896/volumes" Oct 30 17:42:03.687341 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:03.687304 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:42:05.997813 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:42:05.997777 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:42:06.365539 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.365510 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g"] Oct 30 17:42:06.366633 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.366613 4965 cpu_manager.go:414] "containerMap: removing container" podUID="123ee2af-0589-49d5-b1a5-3f68b6ac9432" containerName="istio-proxy" Oct 30 17:42:06.366685 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.366635 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="123ee2af-0589-49d5-b1a5-3f68b6ac9432" containerName="istio-proxy" Oct 30 17:42:06.366685 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.366671 4965 cpu_manager.go:414] "containerMap: removing container" podUID="174ed90c-b452-485c-9dfb-af6a167089b0" containerName="payload-processing" Oct 30 17:42:06.366685 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.366679 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="174ed90c-b452-485c-9dfb-af6a167089b0" containerName="payload-processing" Oct 30 17:42:06.366794 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.366695 4965 cpu_manager.go:414] "containerMap: removing container" podUID="a4c7171d-f08d-484e-8201-2e8644360ede" containerName="istio-proxy" Oct 30 17:42:06.366794 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.366702 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4c7171d-f08d-484e-8201-2e8644360ede" containerName="istio-proxy" Oct 30 17:42:06.366794 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.366713 4965 cpu_manager.go:414] "containerMap: removing container" podUID="09efdc53-164b-4276-8179-5dffd1c33896" containerName="istio-proxy" Oct 30 17:42:06.366794 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.366721 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09efdc53-164b-4276-8179-5dffd1c33896" containerName="istio-proxy" Oct 30 17:42:06.366794 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.366734 4965 cpu_manager.go:414] "containerMap: removing container" podUID="8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d" containerName="istio-proxy" Oct 30 17:42:06.366794 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.366741 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d" containerName="istio-proxy" Oct 30 17:42:06.366794 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.366760 4965 cpu_manager.go:414] "containerMap: removing container" podUID="8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d" containerName="istio-proxy" Oct 30 17:42:06.366794 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.366767 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d" containerName="istio-proxy" Oct 30 17:42:06.366794 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.366785 4965 cpu_manager.go:414] "containerMap: removing container" podUID="9960af67-a570-4962-a0ac-770879a98949" containerName="payload-pre-processing" Oct 30 17:42:06.366794 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.366792 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9960af67-a570-4962-a0ac-770879a98949" containerName="payload-pre-processing" Oct 30 17:42:06.367095 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.366980 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="123ee2af-0589-49d5-b1a5-3f68b6ac9432" containerName="istio-proxy" Oct 30 17:42:06.367095 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.366997 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="174ed90c-b452-485c-9dfb-af6a167089b0" containerName="payload-processing" Oct 30 17:42:06.367095 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.367008 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4c7171d-f08d-484e-8201-2e8644360ede" containerName="istio-proxy" Oct 30 17:42:06.367095 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.367017 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="09efdc53-164b-4276-8179-5dffd1c33896" containerName="istio-proxy" Oct 30 17:42:06.367095 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.367026 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d" containerName="istio-proxy" Oct 30 17:42:06.367095 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.367036 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e4a1789-b3e2-4415-91dd-b9dd1e27cd4d" containerName="istio-proxy" Oct 30 17:42:06.367095 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.367049 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="9960af67-a570-4962-a0ac-770879a98949" containerName="payload-pre-processing" Oct 30 17:42:06.371356 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.371334 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:06.385591 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.385563 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g"] Oct 30 17:42:06.400958 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.392211 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-874b83-openshift-default-dockercfg-nmtqj\"" Oct 30 17:42:06.640940 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.636393 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-credential-socket\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:06.640940 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.636486 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvzps\" (UniqueName: \"kubernetes.io/projected/508ddd74-da86-4674-ad80-7af4a1eb7310-kube-api-access-xvzps\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:06.640940 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.636587 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/508ddd74-da86-4674-ad80-7af4a1eb7310-istiod-ca-cert\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:06.640940 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.636665 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/508ddd74-da86-4674-ad80-7af4a1eb7310-istio-token\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:06.640940 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.636709 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-workload-certs\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:06.640940 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.636802 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-workload-socket\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:06.640940 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.636942 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-istio-data\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:06.640940 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.637097 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/508ddd74-da86-4674-ad80-7af4a1eb7310-istio-podinfo\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:06.640940 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.637178 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-istio-envoy\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:06.744883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.739481 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-credential-socket\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:06.744883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.739925 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-credential-socket\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:06.744883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.740111 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvzps\" (UniqueName: \"kubernetes.io/projected/508ddd74-da86-4674-ad80-7af4a1eb7310-kube-api-access-xvzps\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:06.744883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.740611 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/508ddd74-da86-4674-ad80-7af4a1eb7310-istiod-ca-cert\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:06.744883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.741419 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/508ddd74-da86-4674-ad80-7af4a1eb7310-istiod-ca-cert\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:06.754880 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.751096 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/508ddd74-da86-4674-ad80-7af4a1eb7310-istio-token\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:06.754880 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.751474 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-workload-certs\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:06.754880 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.751781 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-workload-certs\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:06.754880 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.751939 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-workload-socket\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:06.754880 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.752202 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-workload-socket\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:06.754880 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.752486 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-istio-data\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:06.754880 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.752514 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-istio-data\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:06.754880 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.752582 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/508ddd74-da86-4674-ad80-7af4a1eb7310-istio-podinfo\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:06.754880 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.753112 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-istio-envoy\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:06.798888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.789854 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/508ddd74-da86-4674-ad80-7af4a1eb7310-istio-token\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:06.819886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.814247 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/508ddd74-da86-4674-ad80-7af4a1eb7310-istio-podinfo\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:06.819886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.815170 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-istio-envoy\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:06.819886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:06.815467 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvzps\" (UniqueName: \"kubernetes.io/projected/508ddd74-da86-4674-ad80-7af4a1eb7310-kube-api-access-xvzps\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:07.010324 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:07.008578 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:07.566607 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:42:07.566571 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod508ddd74_da86_4674_ad80_7af4a1eb7310.slice/crio-75b1d603821925e8c78af06cfcd168e3c72b7f23907629d4d37582217cb36f92 WatchSource:0}: Error finding container 75b1d603821925e8c78af06cfcd168e3c72b7f23907629d4d37582217cb36f92: Status 404 returned error can't find the container with id 75b1d603821925e8c78af06cfcd168e3c72b7f23907629d4d37582217cb36f92 Oct 30 17:42:07.588194 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:07.587182 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g"] Oct 30 17:42:07.593455 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:07.593380 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:42:07.594892 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:07.593976 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:42:07.594892 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:07.594327 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:42:07.867918 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:07.865243 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" event={"ID":"508ddd74-da86-4674-ad80-7af4a1eb7310","Type":"ContainerStarted","Data":"a00bc22a52abafa1d7649f89563b915129c6562f936fc8ef33227ab8d4aff299"} Oct 30 17:42:07.867918 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:07.865334 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" event={"ID":"508ddd74-da86-4674-ad80-7af4a1eb7310","Type":"ContainerStarted","Data":"75b1d603821925e8c78af06cfcd168e3c72b7f23907629d4d37582217cb36f92"} Oct 30 17:42:07.927230 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:07.927176 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" podStartSLOduration=1.9271622499999999 podStartE2EDuration="1.92716225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:42:07.926140534 +0000 UTC m=+1785.978240487" watchObservedRunningTime="2026-10-30 17:42:07.92716225 +0000 UTC m=+1785.979262183" Oct 30 17:42:08.009968 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:08.009931 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:08.011218 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:08.011196 4965 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 17:42:08.011588 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:08.011254 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" podUID="508ddd74-da86-4674-ad80-7af4a1eb7310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 17:42:09.014468 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:09.014439 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:09.383699 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:09.383667 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9"] Oct 30 17:42:09.388730 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:09.388710 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9" Oct 30 17:42:09.407697 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:09.407673 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9"] Oct 30 17:42:09.450315 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:09.450279 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0c4b63-604c-49de-9f33-4465aec216bc-maas-api-tls\") pod \"maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9\" (UID: \"5a0c4b63-604c-49de-9f33-4465aec216bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9" Oct 30 17:42:09.450501 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:09.450417 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a0c4b63-604c-49de-9f33-4465aec216bc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9\" (UID: \"5a0c4b63-604c-49de-9f33-4465aec216bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9" Oct 30 17:42:09.450564 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:09.450516 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a0c4b63-604c-49de-9f33-4465aec216bc-metrics-certs\") pod \"maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9\" (UID: \"5a0c4b63-604c-49de-9f33-4465aec216bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9" Oct 30 17:42:09.450740 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:09.450708 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns6zw\" (UniqueName: \"kubernetes.io/projected/5a0c4b63-604c-49de-9f33-4465aec216bc-kube-api-access-ns6zw\") pod \"maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9\" (UID: \"5a0c4b63-604c-49de-9f33-4465aec216bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9" Oct 30 17:42:09.476741 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:09.476713 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-bba90b-serving-cert\"" Oct 30 17:42:09.551872 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:09.551827 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a0c4b63-604c-49de-9f33-4465aec216bc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9\" (UID: \"5a0c4b63-604c-49de-9f33-4465aec216bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9" Oct 30 17:42:09.552029 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:09.551926 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a0c4b63-604c-49de-9f33-4465aec216bc-metrics-certs\") pod \"maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9\" (UID: \"5a0c4b63-604c-49de-9f33-4465aec216bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9" Oct 30 17:42:09.552029 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:09.552019 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns6zw\" (UniqueName: \"kubernetes.io/projected/5a0c4b63-604c-49de-9f33-4465aec216bc-kube-api-access-ns6zw\") pod \"maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9\" (UID: \"5a0c4b63-604c-49de-9f33-4465aec216bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9" Oct 30 17:42:09.552158 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:09.552144 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0c4b63-604c-49de-9f33-4465aec216bc-maas-api-tls\") pod \"maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9\" (UID: \"5a0c4b63-604c-49de-9f33-4465aec216bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9" Oct 30 17:42:09.552551 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:09.552523 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a0c4b63-604c-49de-9f33-4465aec216bc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9\" (UID: \"5a0c4b63-604c-49de-9f33-4465aec216bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9" Oct 30 17:42:09.554365 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:09.554339 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a0c4b63-604c-49de-9f33-4465aec216bc-metrics-certs\") pod \"maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9\" (UID: \"5a0c4b63-604c-49de-9f33-4465aec216bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9" Oct 30 17:42:09.554564 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:09.554542 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0c4b63-604c-49de-9f33-4465aec216bc-maas-api-tls\") pod \"maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9\" (UID: \"5a0c4b63-604c-49de-9f33-4465aec216bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9" Oct 30 17:42:09.576169 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:09.576139 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns6zw\" (UniqueName: \"kubernetes.io/projected/5a0c4b63-604c-49de-9f33-4465aec216bc-kube-api-access-ns6zw\") pod \"maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9\" (UID: \"5a0c4b63-604c-49de-9f33-4465aec216bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9" Oct 30 17:42:09.707942 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:09.707190 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9" Oct 30 17:42:09.885197 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:09.885162 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:09.886214 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:09.886190 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:42:10.062950 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:42:10.062921 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a0c4b63_604c_49de_9f33_4465aec216bc.slice/crio-bb9e5104e296298477d59186810d4e4453bdec90259e9598457b77d8a38dd1b7 WatchSource:0}: Error finding container bb9e5104e296298477d59186810d4e4453bdec90259e9598457b77d8a38dd1b7: Status 404 returned error can't find the container with id bb9e5104e296298477d59186810d4e4453bdec90259e9598457b77d8a38dd1b7 Oct 30 17:42:10.063437 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:10.063415 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9"] Oct 30 17:42:10.064761 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:10.064745 4965 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:42:10.462114 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:10.462071 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-802bda-6cbf7f6d44-fnfr8"] Oct 30 17:42:10.467454 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:10.466300 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-802bda-55497ddf87-452b4"] Oct 30 17:42:11.011888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:11.011067 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9" event={"ID":"5a0c4b63-604c-49de-9f33-4465aec216bc","Type":"ContainerStarted","Data":"f7ebed19cc30a426b2e9bf305e85cb35e8eb98cf115c5c50970aa5a5e02b0198"} Oct 30 17:42:11.011888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:11.011138 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9" event={"ID":"5a0c4b63-604c-49de-9f33-4465aec216bc","Type":"ContainerStarted","Data":"bb9e5104e296298477d59186810d4e4453bdec90259e9598457b77d8a38dd1b7"} Oct 30 17:42:11.011888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:11.011623 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9" Oct 30 17:42:11.115925 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:11.115852 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9" podStartSLOduration=1.715863621 podStartE2EDuration="2.115830414s" totalImagesPullingTime="399.966793ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:42:11.066333626 +0000 UTC m=+1789.118433559" watchObservedRunningTime="2026-10-30 17:42:11.115830414 +0000 UTC m=+1789.167930348" Oct 30 17:42:15.462961 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:15.462902 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-802bda-6cbf7f6d44-fnfr8" podUID="704f5086-b986-45c0-b515-a04fd6e0a0e0" containerName="payload-processing" containerID="cri-o://21e34ea2ae10582e477be0edef02dd1ad14bc90c3b1996f5f01181ea4355d4b3" gracePeriod=25 Oct 30 17:42:15.467153 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:15.467106 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-802bda-55497ddf87-452b4" podUID="e9ebc66a-28f5-4a2e-ae0e-28933f360997" containerName="payload-pre-processing" containerID="cri-o://0f8f583735f82b5e8b628fa912144fe3af2c08e87cec45925659d554d858b123" gracePeriod=25 Oct 30 17:42:15.856368 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:15.856341 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-802bda-6cbf7f6d44-fnfr8" Oct 30 17:42:15.921882 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:15.921463 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/704f5086-b986-45c0-b515-a04fd6e0a0e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/704f5086-b986-45c0-b515-a04fd6e0a0e0-plugins-config-volume\") pod \"704f5086-b986-45c0-b515-a04fd6e0a0e0\" (UID: \"704f5086-b986-45c0-b515-a04fd6e0a0e0\") " Oct 30 17:42:15.921882 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:15.921573 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/704f5086-b986-45c0-b515-a04fd6e0a0e0-kube-api-access-82j7w\" (UniqueName: \"kubernetes.io/projected/704f5086-b986-45c0-b515-a04fd6e0a0e0-kube-api-access-82j7w\") pod \"704f5086-b986-45c0-b515-a04fd6e0a0e0\" (UID: \"704f5086-b986-45c0-b515-a04fd6e0a0e0\") " Oct 30 17:42:15.925847 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:15.922849 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/704f5086-b986-45c0-b515-a04fd6e0a0e0-plugins-config-volume" pod "704f5086-b986-45c0-b515-a04fd6e0a0e0" (UID: "704f5086-b986-45c0-b515-a04fd6e0a0e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:42:15.984931 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:15.979396 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-802bda-55497ddf87-452b4" Oct 30 17:42:16.014906 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:16.014116 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/704f5086-b986-45c0-b515-a04fd6e0a0e0-kube-api-access-82j7w" pod "704f5086-b986-45c0-b515-a04fd6e0a0e0" (UID: "704f5086-b986-45c0-b515-a04fd6e0a0e0"). InnerVolumeSpecName "kube-api-access-82j7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:42:16.029885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:16.029768 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/704f5086-b986-45c0-b515-a04fd6e0a0e0-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:42:16.029885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:16.029804 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82j7w\" (UniqueName: \"kubernetes.io/projected/704f5086-b986-45c0-b515-a04fd6e0a0e0-kube-api-access-82j7w\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:42:16.130441 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:16.130406 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9ebc66a-28f5-4a2e-ae0e-28933f360997-kube-api-access-5zt7s\" (UniqueName: \"kubernetes.io/projected/e9ebc66a-28f5-4a2e-ae0e-28933f360997-kube-api-access-5zt7s\") pod \"e9ebc66a-28f5-4a2e-ae0e-28933f360997\" (UID: \"e9ebc66a-28f5-4a2e-ae0e-28933f360997\") " Oct 30 17:42:16.130620 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:16.130606 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9ebc66a-28f5-4a2e-ae0e-28933f360997-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ebc66a-28f5-4a2e-ae0e-28933f360997-plugins-config-volume\") pod \"e9ebc66a-28f5-4a2e-ae0e-28933f360997\" (UID: \"e9ebc66a-28f5-4a2e-ae0e-28933f360997\") " Oct 30 17:42:16.130998 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:16.130973 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ebc66a-28f5-4a2e-ae0e-28933f360997-plugins-config-volume" pod "e9ebc66a-28f5-4a2e-ae0e-28933f360997" (UID: "e9ebc66a-28f5-4a2e-ae0e-28933f360997"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:42:16.131151 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:16.131136 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ebc66a-28f5-4a2e-ae0e-28933f360997-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:42:16.132530 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:16.132506 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ebc66a-28f5-4a2e-ae0e-28933f360997-kube-api-access-5zt7s" pod "e9ebc66a-28f5-4a2e-ae0e-28933f360997" (UID: "e9ebc66a-28f5-4a2e-ae0e-28933f360997"). InnerVolumeSpecName "kube-api-access-5zt7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:42:16.170413 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:16.170384 4965 generic.go:356] "Generic (PLEG): container finished" podID="e9ebc66a-28f5-4a2e-ae0e-28933f360997" containerID="0f8f583735f82b5e8b628fa912144fe3af2c08e87cec45925659d554d858b123" exitCode=0 Oct 30 17:42:16.170523 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:16.170431 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-802bda-55497ddf87-452b4" Oct 30 17:42:16.170523 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:16.170472 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-802bda-55497ddf87-452b4" event={"ID":"e9ebc66a-28f5-4a2e-ae0e-28933f360997","Type":"ContainerDied","Data":"0f8f583735f82b5e8b628fa912144fe3af2c08e87cec45925659d554d858b123"} Oct 30 17:42:16.170613 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:16.170527 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-802bda-55497ddf87-452b4" event={"ID":"e9ebc66a-28f5-4a2e-ae0e-28933f360997","Type":"ContainerDied","Data":"396456f6197f6ce397c100f8fcdd82d5cf5a5d2616c300f584d263710e205d42"} Oct 30 17:42:16.170613 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:16.170554 4965 scope.go:122] "RemoveContainer" containerID="0f8f583735f82b5e8b628fa912144fe3af2c08e87cec45925659d554d858b123" Oct 30 17:42:16.171996 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:16.171975 4965 generic.go:356] "Generic (PLEG): container finished" podID="704f5086-b986-45c0-b515-a04fd6e0a0e0" containerID="21e34ea2ae10582e477be0edef02dd1ad14bc90c3b1996f5f01181ea4355d4b3" exitCode=0 Oct 30 17:42:16.172105 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:16.172032 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-802bda-6cbf7f6d44-fnfr8" Oct 30 17:42:16.172167 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:16.172041 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-802bda-6cbf7f6d44-fnfr8" event={"ID":"704f5086-b986-45c0-b515-a04fd6e0a0e0","Type":"ContainerDied","Data":"21e34ea2ae10582e477be0edef02dd1ad14bc90c3b1996f5f01181ea4355d4b3"} Oct 30 17:42:16.172222 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:16.172163 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-802bda-6cbf7f6d44-fnfr8" event={"ID":"704f5086-b986-45c0-b515-a04fd6e0a0e0","Type":"ContainerDied","Data":"f4265ebe201400dea4a3495a6b54c59c79a3ba4f0f72ff9d12bb9464dd48d5cd"} Oct 30 17:42:16.181584 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:16.181569 4965 scope.go:122] "RemoveContainer" containerID="0f8f583735f82b5e8b628fa912144fe3af2c08e87cec45925659d554d858b123" Oct 30 17:42:16.181870 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:42:16.181843 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f8f583735f82b5e8b628fa912144fe3af2c08e87cec45925659d554d858b123\": container with ID starting with 0f8f583735f82b5e8b628fa912144fe3af2c08e87cec45925659d554d858b123 not found: ID does not exist" containerID="0f8f583735f82b5e8b628fa912144fe3af2c08e87cec45925659d554d858b123" Oct 30 17:42:16.181932 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:16.181877 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f8f583735f82b5e8b628fa912144fe3af2c08e87cec45925659d554d858b123"} err="failed to get container status \"0f8f583735f82b5e8b628fa912144fe3af2c08e87cec45925659d554d858b123\": rpc error: code = NotFound desc = could not find container \"0f8f583735f82b5e8b628fa912144fe3af2c08e87cec45925659d554d858b123\": container with ID starting with 0f8f583735f82b5e8b628fa912144fe3af2c08e87cec45925659d554d858b123 not found: ID does not exist" Oct 30 17:42:16.181932 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:16.181892 4965 scope.go:122] "RemoveContainer" containerID="21e34ea2ae10582e477be0edef02dd1ad14bc90c3b1996f5f01181ea4355d4b3" Oct 30 17:42:16.191503 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:16.191484 4965 scope.go:122] "RemoveContainer" containerID="21e34ea2ae10582e477be0edef02dd1ad14bc90c3b1996f5f01181ea4355d4b3" Oct 30 17:42:16.191750 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:42:16.191734 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21e34ea2ae10582e477be0edef02dd1ad14bc90c3b1996f5f01181ea4355d4b3\": container with ID starting with 21e34ea2ae10582e477be0edef02dd1ad14bc90c3b1996f5f01181ea4355d4b3 not found: ID does not exist" containerID="21e34ea2ae10582e477be0edef02dd1ad14bc90c3b1996f5f01181ea4355d4b3" Oct 30 17:42:16.191796 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:16.191756 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e34ea2ae10582e477be0edef02dd1ad14bc90c3b1996f5f01181ea4355d4b3"} err="failed to get container status \"21e34ea2ae10582e477be0edef02dd1ad14bc90c3b1996f5f01181ea4355d4b3\": rpc error: code = NotFound desc = could not find container \"21e34ea2ae10582e477be0edef02dd1ad14bc90c3b1996f5f01181ea4355d4b3\": container with ID starting with 21e34ea2ae10582e477be0edef02dd1ad14bc90c3b1996f5f01181ea4355d4b3 not found: ID does not exist" Oct 30 17:42:16.229321 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:16.229289 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-802bda-55497ddf87-452b4"] Oct 30 17:42:16.232717 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:16.232693 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zt7s\" (UniqueName: \"kubernetes.io/projected/e9ebc66a-28f5-4a2e-ae0e-28933f360997-kube-api-access-5zt7s\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:42:16.235607 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:16.235567 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-802bda-55497ddf87-452b4"] Oct 30 17:42:16.265163 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:16.265134 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-802bda-6cbf7f6d44-fnfr8"] Oct 30 17:42:16.269473 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:16.269448 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-802bda-6cbf7f6d44-fnfr8"] Oct 30 17:42:17.020247 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:17.020209 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9" podUID="5a0c4b63-604c-49de-9f33-4465aec216bc" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.135:8443/health\": dial tcp 10.132.0.135:8443: connect: connection refused" Oct 30 17:42:18.100513 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:18.100478 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="704f5086-b986-45c0-b515-a04fd6e0a0e0" path="/var/lib/kubelet/pods/704f5086-b986-45c0-b515-a04fd6e0a0e0/volumes" Oct 30 17:42:18.100946 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:18.100931 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9ebc66a-28f5-4a2e-ae0e-28933f360997" path="/var/lib/kubelet/pods/e9ebc66a-28f5-4a2e-ae0e-28933f360997/volumes" Oct 30 17:42:21.212964 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:21.212932 4965 generic.go:356] "Generic (PLEG): container finished" podID="5a0c4b63-604c-49de-9f33-4465aec216bc" containerID="f7ebed19cc30a426b2e9bf305e85cb35e8eb98cf115c5c50970aa5a5e02b0198" exitCode=1 Oct 30 17:42:21.213403 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:21.213009 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9" event={"ID":"5a0c4b63-604c-49de-9f33-4465aec216bc","Type":"ContainerDied","Data":"f7ebed19cc30a426b2e9bf305e85cb35e8eb98cf115c5c50970aa5a5e02b0198"} Oct 30 17:42:21.213481 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:21.213466 4965 scope.go:122] "RemoveContainer" containerID="f7ebed19cc30a426b2e9bf305e85cb35e8eb98cf115c5c50970aa5a5e02b0198" Oct 30 17:42:22.222209 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:22.222168 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9" event={"ID":"5a0c4b63-604c-49de-9f33-4465aec216bc","Type":"ContainerStarted","Data":"4c9f8b30ee25ac24925ed69fe7c6bcebe926e12c41a338835b71fd0e58a3748c"} Oct 30 17:42:22.222602 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:22.222394 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9" Oct 30 17:42:22.392351 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:22.392320 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 17:42:22.393239 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:22.393216 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 17:42:28.271131 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:28.271097 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9" Oct 30 17:42:35.998736 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:42:35.998694 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:42:40.224639 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:40.224579 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2"] Oct 30 17:42:40.226361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:40.225484 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" podUID="232650b5-3cab-4611-8ae6-9504a7ac2f04" containerName="istio-proxy" containerID="cri-o://c65597114ab0a8e14f90974af8295469406297585fe8c86c2fbd41ac86d2fc0c" gracePeriod=30 Oct 30 17:42:45.465480 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.465453 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:42:45.574451 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.574355 4965 generic.go:356] "Generic (PLEG): container finished" podID="232650b5-3cab-4611-8ae6-9504a7ac2f04" containerID="c65597114ab0a8e14f90974af8295469406297585fe8c86c2fbd41ac86d2fc0c" exitCode=0 Oct 30 17:42:45.574451 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.574406 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" Oct 30 17:42:45.574673 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.574441 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" event={"ID":"232650b5-3cab-4611-8ae6-9504a7ac2f04","Type":"ContainerDied","Data":"c65597114ab0a8e14f90974af8295469406297585fe8c86c2fbd41ac86d2fc0c"} Oct 30 17:42:45.574673 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.574517 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2" event={"ID":"232650b5-3cab-4611-8ae6-9504a7ac2f04","Type":"ContainerDied","Data":"9a7eacddcfc2c9d099e4401ceea0ee9035b3edcd1e514428922bf6a76e12bc20"} Oct 30 17:42:45.574673 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.574548 4965 scope.go:122] "RemoveContainer" containerID="c65597114ab0a8e14f90974af8295469406297585fe8c86c2fbd41ac86d2fc0c" Oct 30 17:42:45.588494 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.588461 4965 scope.go:122] "RemoveContainer" containerID="df56035585263205e669e35fc96fa35e9dfd07c82318038fc3d0bc5eabc0cdac" Oct 30 17:42:45.599428 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.599405 4965 scope.go:122] "RemoveContainer" containerID="c65597114ab0a8e14f90974af8295469406297585fe8c86c2fbd41ac86d2fc0c" Oct 30 17:42:45.599709 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:42:45.599691 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c65597114ab0a8e14f90974af8295469406297585fe8c86c2fbd41ac86d2fc0c\": container with ID starting with c65597114ab0a8e14f90974af8295469406297585fe8c86c2fbd41ac86d2fc0c not found: ID does not exist" containerID="c65597114ab0a8e14f90974af8295469406297585fe8c86c2fbd41ac86d2fc0c" Oct 30 17:42:45.599768 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.599717 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c65597114ab0a8e14f90974af8295469406297585fe8c86c2fbd41ac86d2fc0c"} err="failed to get container status \"c65597114ab0a8e14f90974af8295469406297585fe8c86c2fbd41ac86d2fc0c\": rpc error: code = NotFound desc = could not find container \"c65597114ab0a8e14f90974af8295469406297585fe8c86c2fbd41ac86d2fc0c\": container with ID starting with c65597114ab0a8e14f90974af8295469406297585fe8c86c2fbd41ac86d2fc0c not found: ID does not exist" Oct 30 17:42:45.599768 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.599733 4965 scope.go:122] "RemoveContainer" containerID="df56035585263205e669e35fc96fa35e9dfd07c82318038fc3d0bc5eabc0cdac" Oct 30 17:42:45.600095 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:42:45.600040 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df56035585263205e669e35fc96fa35e9dfd07c82318038fc3d0bc5eabc0cdac\": container with ID starting with df56035585263205e669e35fc96fa35e9dfd07c82318038fc3d0bc5eabc0cdac not found: ID does not exist" containerID="df56035585263205e669e35fc96fa35e9dfd07c82318038fc3d0bc5eabc0cdac" Oct 30 17:42:45.600095 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.600074 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df56035585263205e669e35fc96fa35e9dfd07c82318038fc3d0bc5eabc0cdac"} err="failed to get container status \"df56035585263205e669e35fc96fa35e9dfd07c82318038fc3d0bc5eabc0cdac\": rpc error: code = NotFound desc = could not find container \"df56035585263205e669e35fc96fa35e9dfd07c82318038fc3d0bc5eabc0cdac\": container with ID starting with df56035585263205e669e35fc96fa35e9dfd07c82318038fc3d0bc5eabc0cdac not found: ID does not exist" Oct 30 17:42:45.623768 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.623736 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-workload-certs\") pod \"232650b5-3cab-4611-8ae6-9504a7ac2f04\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " Oct 30 17:42:45.623931 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.623792 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/232650b5-3cab-4611-8ae6-9504a7ac2f04-kube-api-access-vhb8r\" (UniqueName: \"kubernetes.io/projected/232650b5-3cab-4611-8ae6-9504a7ac2f04-kube-api-access-vhb8r\") pod \"232650b5-3cab-4611-8ae6-9504a7ac2f04\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " Oct 30 17:42:45.623931 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.623846 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/232650b5-3cab-4611-8ae6-9504a7ac2f04-istio-token\" (UniqueName: \"kubernetes.io/projected/232650b5-3cab-4611-8ae6-9504a7ac2f04-istio-token\") pod \"232650b5-3cab-4611-8ae6-9504a7ac2f04\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " Oct 30 17:42:45.624054 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.624026 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-istio-envoy\") pod \"232650b5-3cab-4611-8ae6-9504a7ac2f04\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " Oct 30 17:42:45.624118 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.624066 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-workload-certs" pod "232650b5-3cab-4611-8ae6-9504a7ac2f04" (UID: "232650b5-3cab-4611-8ae6-9504a7ac2f04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:42:45.624118 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.624074 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/232650b5-3cab-4611-8ae6-9504a7ac2f04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232650b5-3cab-4611-8ae6-9504a7ac2f04-istio-podinfo\") pod \"232650b5-3cab-4611-8ae6-9504a7ac2f04\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " Oct 30 17:42:45.624220 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.624157 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-istio-data\") pod \"232650b5-3cab-4611-8ae6-9504a7ac2f04\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " Oct 30 17:42:45.624220 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.624209 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-credential-socket\") pod \"232650b5-3cab-4611-8ae6-9504a7ac2f04\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " Oct 30 17:42:45.624335 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.624268 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/232650b5-3cab-4611-8ae6-9504a7ac2f04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232650b5-3cab-4611-8ae6-9504a7ac2f04-istiod-ca-cert\") pod \"232650b5-3cab-4611-8ae6-9504a7ac2f04\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " Oct 30 17:42:45.624456 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.624434 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-istio-data" pod "232650b5-3cab-4611-8ae6-9504a7ac2f04" (UID: "232650b5-3cab-4611-8ae6-9504a7ac2f04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:42:45.624518 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.624467 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-credential-socket" pod "232650b5-3cab-4611-8ae6-9504a7ac2f04" (UID: "232650b5-3cab-4611-8ae6-9504a7ac2f04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:42:45.624561 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.624542 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-workload-socket\") pod \"232650b5-3cab-4611-8ae6-9504a7ac2f04\" (UID: \"232650b5-3cab-4611-8ae6-9504a7ac2f04\") " Oct 30 17:42:45.624718 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.624690 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/232650b5-3cab-4611-8ae6-9504a7ac2f04-istiod-ca-cert" pod "232650b5-3cab-4611-8ae6-9504a7ac2f04" (UID: "232650b5-3cab-4611-8ae6-9504a7ac2f04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:42:45.624816 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.624773 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-workload-socket" pod "232650b5-3cab-4611-8ae6-9504a7ac2f04" (UID: "232650b5-3cab-4611-8ae6-9504a7ac2f04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:42:45.625550 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.625527 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:42:45.625605 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.625558 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:42:45.625605 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.625572 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232650b5-3cab-4611-8ae6-9504a7ac2f04-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:42:45.625605 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.625586 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:42:45.625605 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.625598 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:42:45.626316 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.626294 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/232650b5-3cab-4611-8ae6-9504a7ac2f04-istio-podinfo" pod "232650b5-3cab-4611-8ae6-9504a7ac2f04" (UID: "232650b5-3cab-4611-8ae6-9504a7ac2f04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:42:45.626518 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.626496 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/232650b5-3cab-4611-8ae6-9504a7ac2f04-kube-api-access-vhb8r" pod "232650b5-3cab-4611-8ae6-9504a7ac2f04" (UID: "232650b5-3cab-4611-8ae6-9504a7ac2f04"). InnerVolumeSpecName "kube-api-access-vhb8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:42:45.626646 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.626622 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-istio-envoy" pod "232650b5-3cab-4611-8ae6-9504a7ac2f04" (UID: "232650b5-3cab-4611-8ae6-9504a7ac2f04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:42:45.626686 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.626670 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/232650b5-3cab-4611-8ae6-9504a7ac2f04-istio-token" pod "232650b5-3cab-4611-8ae6-9504a7ac2f04" (UID: "232650b5-3cab-4611-8ae6-9504a7ac2f04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:42:45.727126 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.727087 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhb8r\" (UniqueName: \"kubernetes.io/projected/232650b5-3cab-4611-8ae6-9504a7ac2f04-kube-api-access-vhb8r\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:42:45.727126 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.727120 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232650b5-3cab-4611-8ae6-9504a7ac2f04-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:42:45.727126 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.727129 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232650b5-3cab-4611-8ae6-9504a7ac2f04-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:42:45.727347 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.727136 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232650b5-3cab-4611-8ae6-9504a7ac2f04-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:42:45.954153 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.954092 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2"] Oct 30 17:42:45.961909 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:45.961878 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-bltw2"] Oct 30 17:42:46.108188 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:46.108153 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="232650b5-3cab-4611-8ae6-9504a7ac2f04" path="/var/lib/kubelet/pods/232650b5-3cab-4611-8ae6-9504a7ac2f04/volumes" Oct 30 17:42:46.210138 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:46.210059 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g"] Oct 30 17:42:46.210376 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:46.210324 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" podUID="30e31aef-c00b-4418-b005-50b3689d2912" containerName="istio-proxy" containerID="cri-o://8e1f255523cf65c0a4396d77af9241f02a140f656c67ba7038fd92dacde165d0" gracePeriod=30 Oct 30 17:42:47.472663 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:47.472632 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62"] Oct 30 17:42:47.473441 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:47.473425 4965 cpu_manager.go:414] "containerMap: removing container" podUID="704f5086-b986-45c0-b515-a04fd6e0a0e0" containerName="payload-processing" Oct 30 17:42:47.473486 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:47.473443 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="704f5086-b986-45c0-b515-a04fd6e0a0e0" containerName="payload-processing" Oct 30 17:42:47.473486 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:47.473463 4965 cpu_manager.go:414] "containerMap: removing container" podUID="232650b5-3cab-4611-8ae6-9504a7ac2f04" containerName="istio-proxy" Oct 30 17:42:47.473486 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:47.473468 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="232650b5-3cab-4611-8ae6-9504a7ac2f04" containerName="istio-proxy" Oct 30 17:42:47.473587 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:47.473492 4965 cpu_manager.go:414] "containerMap: removing container" podUID="e9ebc66a-28f5-4a2e-ae0e-28933f360997" containerName="payload-pre-processing" Oct 30 17:42:47.473587 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:47.473498 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9ebc66a-28f5-4a2e-ae0e-28933f360997" containerName="payload-pre-processing" Oct 30 17:42:47.473654 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:47.473614 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="232650b5-3cab-4611-8ae6-9504a7ac2f04" containerName="istio-proxy" Oct 30 17:42:47.473654 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:47.473623 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="704f5086-b986-45c0-b515-a04fd6e0a0e0" containerName="payload-processing" Oct 30 17:42:47.473654 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:47.473633 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="232650b5-3cab-4611-8ae6-9504a7ac2f04" containerName="istio-proxy" Oct 30 17:42:47.473654 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:47.473642 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9ebc66a-28f5-4a2e-ae0e-28933f360997" containerName="payload-pre-processing" Oct 30 17:42:47.478959 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:47.478939 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62" Oct 30 17:42:47.497628 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:47.497599 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62"] Oct 30 17:42:47.506702 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:47.506677 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-874b83-serving-cert\"" Oct 30 17:42:47.575430 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:47.575390 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca-metrics-certs\") pod \"maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62\" (UID: \"cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62" Oct 30 17:42:47.575658 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:47.575508 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca-maas-api-tls\") pod \"maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62\" (UID: \"cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62" Oct 30 17:42:47.575658 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:47.575601 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsvlw\" (UniqueName: \"kubernetes.io/projected/cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca-kube-api-access-qsvlw\") pod \"maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62\" (UID: \"cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62" Oct 30 17:42:47.575814 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:47.575696 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca-openshift-service-ca\") pod \"maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62\" (UID: \"cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62" Oct 30 17:42:47.677384 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:47.677352 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca-openshift-service-ca\") pod \"maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62\" (UID: \"cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62" Oct 30 17:42:47.677592 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:47.677499 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca-metrics-certs\") pod \"maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62\" (UID: \"cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62" Oct 30 17:42:47.677669 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:47.677617 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca-maas-api-tls\") pod \"maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62\" (UID: \"cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62" Oct 30 17:42:47.677774 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:47.677756 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsvlw\" (UniqueName: \"kubernetes.io/projected/cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca-kube-api-access-qsvlw\") pod \"maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62\" (UID: \"cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62" Oct 30 17:42:47.678094 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:47.678069 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca-openshift-service-ca\") pod \"maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62\" (UID: \"cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62" Oct 30 17:42:47.680032 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:47.680011 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca-maas-api-tls\") pod \"maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62\" (UID: \"cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62" Oct 30 17:42:47.680426 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:47.680405 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca-metrics-certs\") pod \"maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62\" (UID: \"cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62" Oct 30 17:42:47.702610 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:47.702587 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsvlw\" (UniqueName: \"kubernetes.io/projected/cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca-kube-api-access-qsvlw\") pod \"maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62\" (UID: \"cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62" Oct 30 17:42:47.794522 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:47.793764 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62" Oct 30 17:42:48.188457 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:42:48.188427 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf2a6d30_2a2c_4ee2_a4ad_18d917a634ca.slice/crio-feb3efef6cddd9cebbf0c9084d6a198c7fdce678853c160fcc523a7a06c6b6cc WatchSource:0}: Error finding container feb3efef6cddd9cebbf0c9084d6a198c7fdce678853c160fcc523a7a06c6b6cc: Status 404 returned error can't find the container with id feb3efef6cddd9cebbf0c9084d6a198c7fdce678853c160fcc523a7a06c6b6cc Oct 30 17:42:48.188922 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:48.188897 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62"] Oct 30 17:42:48.317083 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:48.317050 4965 patch_prober.go:29] interesting pod/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:42:48.317253 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:48.317102 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" podUID="30e31aef-c00b-4418-b005-50b3689d2912" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:42:48.677836 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:48.677793 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62" event={"ID":"cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca","Type":"ContainerStarted","Data":"f1cebd30075e3e6bf733a83dafa413f1bf6128b2005b3cbb19a87a2e53c355fc"} Oct 30 17:42:48.678524 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:48.677890 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62" event={"ID":"cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca","Type":"ContainerStarted","Data":"feb3efef6cddd9cebbf0c9084d6a198c7fdce678853c160fcc523a7a06c6b6cc"} Oct 30 17:42:48.679100 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:48.679075 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62" Oct 30 17:42:48.728291 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:48.728180 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62" podStartSLOduration=1.451264772 podStartE2EDuration="1.728161477s" totalImagesPullingTime="276.896705ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:42:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:42:48.725078596 +0000 UTC m=+1826.777178528" watchObservedRunningTime="2026-10-30 17:42:48.728161477 +0000 UTC m=+1826.780261409" Oct 30 17:42:51.524263 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.524236 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:42:51.561602 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.561561 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-istio-data\") pod \"30e31aef-c00b-4418-b005-50b3689d2912\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " Oct 30 17:42:51.561771 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.561714 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-istio-envoy\") pod \"30e31aef-c00b-4418-b005-50b3689d2912\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " Oct 30 17:42:51.561771 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.561748 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30e31aef-c00b-4418-b005-50b3689d2912-kube-api-access-76wnm\" (UniqueName: \"kubernetes.io/projected/30e31aef-c00b-4418-b005-50b3689d2912-kube-api-access-76wnm\") pod \"30e31aef-c00b-4418-b005-50b3689d2912\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " Oct 30 17:42:51.561962 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.561773 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-credential-socket\") pod \"30e31aef-c00b-4418-b005-50b3689d2912\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " Oct 30 17:42:51.561962 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.561855 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-istio-data" pod "30e31aef-c00b-4418-b005-50b3689d2912" (UID: "30e31aef-c00b-4418-b005-50b3689d2912"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:42:51.561962 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.561917 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30e31aef-c00b-4418-b005-50b3689d2912-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30e31aef-c00b-4418-b005-50b3689d2912-istio-podinfo\") pod \"30e31aef-c00b-4418-b005-50b3689d2912\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " Oct 30 17:42:51.562134 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.562009 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-workload-socket\") pod \"30e31aef-c00b-4418-b005-50b3689d2912\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " Oct 30 17:42:51.562190 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.562130 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-credential-socket" pod "30e31aef-c00b-4418-b005-50b3689d2912" (UID: "30e31aef-c00b-4418-b005-50b3689d2912"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:42:51.562190 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.562142 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-workload-certs\") pod \"30e31aef-c00b-4418-b005-50b3689d2912\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " Oct 30 17:42:51.562190 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.562180 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30e31aef-c00b-4418-b005-50b3689d2912-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30e31aef-c00b-4418-b005-50b3689d2912-istiod-ca-cert\") pod \"30e31aef-c00b-4418-b005-50b3689d2912\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " Oct 30 17:42:51.562360 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.562207 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30e31aef-c00b-4418-b005-50b3689d2912-istio-token\" (UniqueName: \"kubernetes.io/projected/30e31aef-c00b-4418-b005-50b3689d2912-istio-token\") pod \"30e31aef-c00b-4418-b005-50b3689d2912\" (UID: \"30e31aef-c00b-4418-b005-50b3689d2912\") " Oct 30 17:42:51.562360 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.562263 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-workload-socket" pod "30e31aef-c00b-4418-b005-50b3689d2912" (UID: "30e31aef-c00b-4418-b005-50b3689d2912"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:42:51.562472 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.562452 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-workload-certs" pod "30e31aef-c00b-4418-b005-50b3689d2912" (UID: "30e31aef-c00b-4418-b005-50b3689d2912"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:42:51.562563 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.562539 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30e31aef-c00b-4418-b005-50b3689d2912-istiod-ca-cert" pod "30e31aef-c00b-4418-b005-50b3689d2912" (UID: "30e31aef-c00b-4418-b005-50b3689d2912"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:42:51.563010 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.562988 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:42:51.563107 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.563012 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:42:51.563107 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.563025 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30e31aef-c00b-4418-b005-50b3689d2912-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:42:51.563107 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.563037 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:42:51.563107 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.563048 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:42:51.564029 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.564007 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-istio-envoy" pod "30e31aef-c00b-4418-b005-50b3689d2912" (UID: "30e31aef-c00b-4418-b005-50b3689d2912"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:42:51.564136 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.564119 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30e31aef-c00b-4418-b005-50b3689d2912-istio-podinfo" pod "30e31aef-c00b-4418-b005-50b3689d2912" (UID: "30e31aef-c00b-4418-b005-50b3689d2912"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:42:51.564367 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.564353 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e31aef-c00b-4418-b005-50b3689d2912-kube-api-access-76wnm" pod "30e31aef-c00b-4418-b005-50b3689d2912" (UID: "30e31aef-c00b-4418-b005-50b3689d2912"). InnerVolumeSpecName "kube-api-access-76wnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:42:51.564547 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.564526 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e31aef-c00b-4418-b005-50b3689d2912-istio-token" pod "30e31aef-c00b-4418-b005-50b3689d2912" (UID: "30e31aef-c00b-4418-b005-50b3689d2912"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:42:51.665031 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.664998 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30e31aef-c00b-4418-b005-50b3689d2912-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:42:51.665031 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.665025 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76wnm\" (UniqueName: \"kubernetes.io/projected/30e31aef-c00b-4418-b005-50b3689d2912-kube-api-access-76wnm\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:42:51.665031 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.665036 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30e31aef-c00b-4418-b005-50b3689d2912-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:42:51.665316 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.665044 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30e31aef-c00b-4418-b005-50b3689d2912-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:42:51.983353 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.982691 4965 generic.go:356] "Generic (PLEG): container finished" podID="30e31aef-c00b-4418-b005-50b3689d2912" containerID="8e1f255523cf65c0a4396d77af9241f02a140f656c67ba7038fd92dacde165d0" exitCode=0 Oct 30 17:42:51.983353 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.982907 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" Oct 30 17:42:51.983353 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.982980 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" event={"ID":"30e31aef-c00b-4418-b005-50b3689d2912","Type":"ContainerDied","Data":"8e1f255523cf65c0a4396d77af9241f02a140f656c67ba7038fd92dacde165d0"} Oct 30 17:42:51.983353 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.983026 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g" event={"ID":"30e31aef-c00b-4418-b005-50b3689d2912","Type":"ContainerDied","Data":"f7ee32cb2ebdf37b8bc4cb83924db1c64b4328b23bf3328e73161b3096e63078"} Oct 30 17:42:51.983353 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:51.983056 4965 scope.go:122] "RemoveContainer" containerID="8e1f255523cf65c0a4396d77af9241f02a140f656c67ba7038fd92dacde165d0" Oct 30 17:42:52.016880 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.016547 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv"] Oct 30 17:42:52.017979 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.017823 4965 cpu_manager.go:414] "containerMap: removing container" podUID="30e31aef-c00b-4418-b005-50b3689d2912" containerName="istio-proxy" Oct 30 17:42:52.017979 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.017849 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30e31aef-c00b-4418-b005-50b3689d2912" containerName="istio-proxy" Oct 30 17:42:52.017979 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.017892 4965 cpu_manager.go:414] "containerMap: removing container" podUID="30e31aef-c00b-4418-b005-50b3689d2912" containerName="istio-proxy" Oct 30 17:42:52.017979 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.017900 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30e31aef-c00b-4418-b005-50b3689d2912" containerName="istio-proxy" Oct 30 17:42:52.017979 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.017917 4965 cpu_manager.go:414] "containerMap: removing container" podUID="232650b5-3cab-4611-8ae6-9504a7ac2f04" containerName="istio-proxy" Oct 30 17:42:52.018320 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.017925 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="232650b5-3cab-4611-8ae6-9504a7ac2f04" containerName="istio-proxy" Oct 30 17:42:52.018320 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.018278 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="30e31aef-c00b-4418-b005-50b3689d2912" containerName="istio-proxy" Oct 30 17:42:52.018320 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.018296 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="30e31aef-c00b-4418-b005-50b3689d2912" containerName="istio-proxy" Oct 30 17:42:52.024836 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.024808 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv" Oct 30 17:42:52.043457 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.043248 4965 scope.go:122] "RemoveContainer" containerID="73f7e236f32f6ded0e7e16ad8dffebf7b745fef40661d5a5517477120842b5f0" Oct 30 17:42:52.043457 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.043320 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-f8b48d-serving-cert\"" Oct 30 17:42:52.043658 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.043518 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv"] Oct 30 17:42:52.061717 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.061672 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g"] Oct 30 17:42:52.062987 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.062967 4965 scope.go:122] "RemoveContainer" containerID="8e1f255523cf65c0a4396d77af9241f02a140f656c67ba7038fd92dacde165d0" Oct 30 17:42:52.064573 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.064525 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c9bdd9-openshift-default-8698669cf-2598g"] Oct 30 17:42:52.064740 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:42:52.064662 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e1f255523cf65c0a4396d77af9241f02a140f656c67ba7038fd92dacde165d0\": container with ID starting with 8e1f255523cf65c0a4396d77af9241f02a140f656c67ba7038fd92dacde165d0 not found: ID does not exist" containerID="8e1f255523cf65c0a4396d77af9241f02a140f656c67ba7038fd92dacde165d0" Oct 30 17:42:52.064948 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.064917 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e1f255523cf65c0a4396d77af9241f02a140f656c67ba7038fd92dacde165d0"} err="failed to get container status \"8e1f255523cf65c0a4396d77af9241f02a140f656c67ba7038fd92dacde165d0\": rpc error: code = NotFound desc = could not find container \"8e1f255523cf65c0a4396d77af9241f02a140f656c67ba7038fd92dacde165d0\": container with ID starting with 8e1f255523cf65c0a4396d77af9241f02a140f656c67ba7038fd92dacde165d0 not found: ID does not exist" Oct 30 17:42:52.065084 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.065068 4965 scope.go:122] "RemoveContainer" containerID="73f7e236f32f6ded0e7e16ad8dffebf7b745fef40661d5a5517477120842b5f0" Oct 30 17:42:52.065558 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:42:52.065535 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73f7e236f32f6ded0e7e16ad8dffebf7b745fef40661d5a5517477120842b5f0\": container with ID starting with 73f7e236f32f6ded0e7e16ad8dffebf7b745fef40661d5a5517477120842b5f0 not found: ID does not exist" containerID="73f7e236f32f6ded0e7e16ad8dffebf7b745fef40661d5a5517477120842b5f0" Oct 30 17:42:52.065748 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.065722 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f7e236f32f6ded0e7e16ad8dffebf7b745fef40661d5a5517477120842b5f0"} err="failed to get container status \"73f7e236f32f6ded0e7e16ad8dffebf7b745fef40661d5a5517477120842b5f0\": rpc error: code = NotFound desc = could not find container \"73f7e236f32f6ded0e7e16ad8dffebf7b745fef40661d5a5517477120842b5f0\": container with ID starting with 73f7e236f32f6ded0e7e16ad8dffebf7b745fef40661d5a5517477120842b5f0 not found: ID does not exist" Oct 30 17:42:52.066620 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.066566 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001-maas-api-tls\") pod \"maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv\" (UID: \"0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv" Oct 30 17:42:52.066728 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.066665 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfkvr\" (UniqueName: \"kubernetes.io/projected/0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001-kube-api-access-sfkvr\") pod \"maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv\" (UID: \"0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv" Oct 30 17:42:52.066958 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.066918 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv\" (UID: \"0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv" Oct 30 17:42:52.067065 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.066978 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001-metrics-certs\") pod \"maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv\" (UID: \"0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv" Oct 30 17:42:52.105652 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.105618 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30e31aef-c00b-4418-b005-50b3689d2912" path="/var/lib/kubelet/pods/30e31aef-c00b-4418-b005-50b3689d2912/volumes" Oct 30 17:42:52.168739 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.168703 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001-metrics-certs\") pod \"maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv\" (UID: \"0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv" Oct 30 17:42:52.168990 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.168970 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001-maas-api-tls\") pod \"maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv\" (UID: \"0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv" Oct 30 17:42:52.169099 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.169067 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfkvr\" (UniqueName: \"kubernetes.io/projected/0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001-kube-api-access-sfkvr\") pod \"maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv\" (UID: \"0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv" Oct 30 17:42:52.169306 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.169284 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv\" (UID: \"0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv" Oct 30 17:42:52.169855 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.169830 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv\" (UID: \"0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv" Oct 30 17:42:52.171179 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.171156 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001-metrics-certs\") pod \"maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv\" (UID: \"0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv" Oct 30 17:42:52.171326 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.171308 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001-maas-api-tls\") pod \"maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv\" (UID: \"0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv" Oct 30 17:42:52.192323 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.192299 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfkvr\" (UniqueName: \"kubernetes.io/projected/0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001-kube-api-access-sfkvr\") pod \"maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv\" (UID: \"0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv" Oct 30 17:42:52.355564 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.355522 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv" Oct 30 17:42:52.389165 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.389135 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m"] Oct 30 17:42:52.396062 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.395812 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:42:52.418272 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.418230 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-74762d-openshift-default-dockercfg-92brx\"" Oct 30 17:42:52.420937 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.420912 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m"] Oct 30 17:42:52.478132 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.478095 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csbkw\" (UniqueName: \"kubernetes.io/projected/1114e4ad-fc83-4876-9ffe-c47afce03fc5-kube-api-access-csbkw\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:42:52.478311 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.478144 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-workload-socket\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:42:52.478311 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.478224 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istio-envoy\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:42:52.478450 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.478326 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istiod-ca-cert\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:42:52.478450 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.478377 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istio-podinfo\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:42:52.478450 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.478396 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istio-token\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:42:52.478450 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.478432 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-workload-certs\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:42:52.478663 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.478563 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-credential-socket\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:42:52.478663 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.478621 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istio-data\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:42:52.579690 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.579652 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istio-data\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:42:52.580177 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.579766 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csbkw\" (UniqueName: \"kubernetes.io/projected/1114e4ad-fc83-4876-9ffe-c47afce03fc5-kube-api-access-csbkw\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:42:52.580177 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.579795 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-workload-socket\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:42:52.580177 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.579814 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istio-envoy\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:42:52.580177 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.579887 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istiod-ca-cert\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:42:52.580177 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.579951 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istio-podinfo\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:42:52.580177 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.579980 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istio-token\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:42:52.580177 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.580076 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istio-data\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:42:52.580177 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.580137 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-workload-certs\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:42:52.580595 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.580190 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-workload-socket\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:42:52.580595 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.580212 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-credential-socket\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:42:52.580595 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.580492 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-credential-socket\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:42:52.580595 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.580489 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-workload-certs\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:42:52.580791 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.580690 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istiod-ca-cert\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:42:52.582319 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.582293 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istio-envoy\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:42:52.582666 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.582647 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istio-podinfo\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:42:52.606011 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.605941 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csbkw\" (UniqueName: \"kubernetes.io/projected/1114e4ad-fc83-4876-9ffe-c47afce03fc5-kube-api-access-csbkw\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:42:52.606327 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.606306 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istio-token\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:42:52.679897 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.679841 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv"] Oct 30 17:42:52.680051 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:42:52.680027 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c26ab04_4bf8_4f0f_bb42_5fcb6b8f5001.slice/crio-36ffd51fd5f3a07ad4a6a1aee8052198ca0cc7d2ee7f02d371cdf917edda95be WatchSource:0}: Error finding container 36ffd51fd5f3a07ad4a6a1aee8052198ca0cc7d2ee7f02d371cdf917edda95be: Status 404 returned error can't find the container with id 36ffd51fd5f3a07ad4a6a1aee8052198ca0cc7d2ee7f02d371cdf917edda95be Oct 30 17:42:52.720926 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.720832 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:42:52.994412 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:52.994330 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv" event={"ID":"0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001","Type":"ContainerStarted","Data":"36ffd51fd5f3a07ad4a6a1aee8052198ca0cc7d2ee7f02d371cdf917edda95be"} Oct 30 17:42:53.083173 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:53.083142 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m"] Oct 30 17:42:53.087727 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:53.087679 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:42:53.087849 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:53.087821 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:42:53.087945 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:53.087920 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:42:54.016504 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:54.016459 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv" event={"ID":"0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001","Type":"ContainerStarted","Data":"8b7d65e75e1c6837b6b492b60da543b38e3eafda2dfefd3841ad534038337b79"} Oct 30 17:42:54.017089 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:54.016540 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv" Oct 30 17:42:54.018665 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:54.018640 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" event={"ID":"1114e4ad-fc83-4876-9ffe-c47afce03fc5","Type":"ContainerStarted","Data":"979d62e7a612b5159e8d58b445165dd65de5af122cce7c42216d87e0d8c8f022"} Oct 30 17:42:54.018807 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:54.018685 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" event={"ID":"1114e4ad-fc83-4876-9ffe-c47afce03fc5","Type":"ContainerStarted","Data":"ecd40e50b6b884a8e489252bc8dd38ad38414465a526858756cef9925a39ee13"} Oct 30 17:42:54.068884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:54.066411 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv" podStartSLOduration=2.675637697 podStartE2EDuration="3.066392575s" totalImagesPullingTime="390.754878ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:42:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:42:54.063336717 +0000 UTC m=+1832.115436650" watchObservedRunningTime="2026-10-30 17:42:54.066392575 +0000 UTC m=+1832.118492514" Oct 30 17:42:54.115827 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:54.115765 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" podStartSLOduration=2.115745812 podStartE2EDuration="2.115745812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:42:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:42:54.110844508 +0000 UTC m=+1832.162944465" watchObservedRunningTime="2026-10-30 17:42:54.115745812 +0000 UTC m=+1832.167845743" Oct 30 17:42:54.723241 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:54.722784 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:42:54.723241 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:54.722927 4965 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 17:42:54.723241 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:54.722972 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" podUID="1114e4ad-fc83-4876-9ffe-c47afce03fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 17:42:55.699579 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:55.699541 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62" Oct 30 17:42:55.721475 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:55.721443 4965 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 17:42:55.721635 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:55.721491 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" podUID="1114e4ad-fc83-4876-9ffe-c47afce03fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 17:42:56.732890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:56.726235 4965 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 17:42:56.732890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:56.726307 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" podUID="1114e4ad-fc83-4876-9ffe-c47afce03fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 17:42:57.380484 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:57.380444 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-49e155-5997c8f79-vqj2r"] Oct 30 17:42:57.731943 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:57.725277 4965 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 17:42:57.731943 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:57.725339 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" podUID="1114e4ad-fc83-4876-9ffe-c47afce03fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 17:42:58.721634 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:58.721598 4965 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 17:42:58.722092 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:58.721655 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" podUID="1114e4ad-fc83-4876-9ffe-c47afce03fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 17:42:59.722218 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:59.722182 4965 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 17:42:59.722608 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:42:59.722270 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" podUID="1114e4ad-fc83-4876-9ffe-c47afce03fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 17:43:00.031614 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:00.031541 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv" Oct 30 17:43:00.735095 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:00.735061 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:43:00.737277 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:00.737254 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:43:00.738775 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:00.738759 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:43:02.126799 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:02.126766 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-143041-6bfddb8867-5tnjk"] Oct 30 17:43:02.132625 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:02.132605 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-143041-6bfddb8867-5tnjk" Oct 30 17:43:02.155258 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:02.155229 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-143041-6bfddb8867-5tnjk"] Oct 30 17:43:02.159614 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:02.159588 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-143041\"" Oct 30 17:43:02.240906 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:02.227208 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-143041-dockercfg-9mz9w\"" Oct 30 17:43:02.391936 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:02.382942 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49e155-5997c8f79-vqj2r" podUID="112bfc33-9359-4f17-be86-f3ffd6ecae0f" containerName="payload-pre-processing" containerID="cri-o://4669f588933210f6b5779f12187f150405915dd20395c6f7c38d3e7bb290ba09" gracePeriod=25 Oct 30 17:43:02.391936 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:02.384220 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59kpx\" (UniqueName: \"kubernetes.io/projected/f25e38d6-2f5a-4b43-bf0e-630706a343a1-kube-api-access-59kpx\") pod \"payload-pre-processing-e2e-inf-b-143041-6bfddb8867-5tnjk\" (UID: \"f25e38d6-2f5a-4b43-bf0e-630706a343a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-143041-6bfddb8867-5tnjk" Oct 30 17:43:02.391936 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:02.384671 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f25e38d6-2f5a-4b43-bf0e-630706a343a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-143041-6bfddb8867-5tnjk\" (UID: \"f25e38d6-2f5a-4b43-bf0e-630706a343a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-143041-6bfddb8867-5tnjk" Oct 30 17:43:02.521035 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:02.489599 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f25e38d6-2f5a-4b43-bf0e-630706a343a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-143041-6bfddb8867-5tnjk\" (UID: \"f25e38d6-2f5a-4b43-bf0e-630706a343a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-143041-6bfddb8867-5tnjk" Oct 30 17:43:02.527994 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:02.527959 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59kpx\" (UniqueName: \"kubernetes.io/projected/f25e38d6-2f5a-4b43-bf0e-630706a343a1-kube-api-access-59kpx\") pod \"payload-pre-processing-e2e-inf-b-143041-6bfddb8867-5tnjk\" (UID: \"f25e38d6-2f5a-4b43-bf0e-630706a343a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-143041-6bfddb8867-5tnjk" Oct 30 17:43:02.558045 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:02.490841 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f25e38d6-2f5a-4b43-bf0e-630706a343a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-143041-6bfddb8867-5tnjk\" (UID: \"f25e38d6-2f5a-4b43-bf0e-630706a343a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-143041-6bfddb8867-5tnjk" Oct 30 17:43:02.605993 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:02.605956 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59kpx\" (UniqueName: \"kubernetes.io/projected/f25e38d6-2f5a-4b43-bf0e-630706a343a1-kube-api-access-59kpx\") pod \"payload-pre-processing-e2e-inf-b-143041-6bfddb8867-5tnjk\" (UID: \"f25e38d6-2f5a-4b43-bf0e-630706a343a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-143041-6bfddb8867-5tnjk" Oct 30 17:43:02.748241 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:02.748198 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-143041-6bfddb8867-5tnjk" Oct 30 17:43:02.805423 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:02.805377 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49e155-5997c8f79-vqj2r" Oct 30 17:43:02.964162 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:02.964048 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/112bfc33-9359-4f17-be86-f3ffd6ecae0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/112bfc33-9359-4f17-be86-f3ffd6ecae0f-plugins-config-volume\") pod \"112bfc33-9359-4f17-be86-f3ffd6ecae0f\" (UID: \"112bfc33-9359-4f17-be86-f3ffd6ecae0f\") " Oct 30 17:43:02.964330 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:02.964246 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/112bfc33-9359-4f17-be86-f3ffd6ecae0f-kube-api-access-dxwvt\" (UniqueName: \"kubernetes.io/projected/112bfc33-9359-4f17-be86-f3ffd6ecae0f-kube-api-access-dxwvt\") pod \"112bfc33-9359-4f17-be86-f3ffd6ecae0f\" (UID: \"112bfc33-9359-4f17-be86-f3ffd6ecae0f\") " Oct 30 17:43:02.964496 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:02.964470 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/112bfc33-9359-4f17-be86-f3ffd6ecae0f-plugins-config-volume" pod "112bfc33-9359-4f17-be86-f3ffd6ecae0f" (UID: "112bfc33-9359-4f17-be86-f3ffd6ecae0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:43:02.964953 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:02.964937 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/112bfc33-9359-4f17-be86-f3ffd6ecae0f-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:43:02.966358 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:02.966334 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/112bfc33-9359-4f17-be86-f3ffd6ecae0f-kube-api-access-dxwvt" pod "112bfc33-9359-4f17-be86-f3ffd6ecae0f" (UID: "112bfc33-9359-4f17-be86-f3ffd6ecae0f"). InnerVolumeSpecName "kube-api-access-dxwvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:43:03.065006 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:03.064954 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-143041-6bfddb8867-5tnjk"] Oct 30 17:43:03.065973 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:03.065953 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxwvt\" (UniqueName: \"kubernetes.io/projected/112bfc33-9359-4f17-be86-f3ffd6ecae0f-kube-api-access-dxwvt\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:43:03.070896 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:43:03.070870 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf25e38d6_2f5a_4b43_bf0e_630706a343a1.slice/crio-b4d82cad31336923e21e6f7a75ec77e542b8a880ca04dc5f31c6a748b873fcde WatchSource:0}: Error finding container b4d82cad31336923e21e6f7a75ec77e542b8a880ca04dc5f31c6a748b873fcde: Status 404 returned error can't find the container with id b4d82cad31336923e21e6f7a75ec77e542b8a880ca04dc5f31c6a748b873fcde Oct 30 17:43:03.261771 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:03.261692 4965 generic.go:356] "Generic (PLEG): container finished" podID="112bfc33-9359-4f17-be86-f3ffd6ecae0f" containerID="4669f588933210f6b5779f12187f150405915dd20395c6f7c38d3e7bb290ba09" exitCode=0 Oct 30 17:43:03.262205 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:03.261756 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49e155-5997c8f79-vqj2r" event={"ID":"112bfc33-9359-4f17-be86-f3ffd6ecae0f","Type":"ContainerDied","Data":"4669f588933210f6b5779f12187f150405915dd20395c6f7c38d3e7bb290ba09"} Oct 30 17:43:03.262205 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:03.261990 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49e155-5997c8f79-vqj2r" event={"ID":"112bfc33-9359-4f17-be86-f3ffd6ecae0f","Type":"ContainerDied","Data":"bbfab169275c313eae811bc9ad11f17676baaec4e825e008c4248f463165fb25"} Oct 30 17:43:03.262205 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:03.261840 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49e155-5997c8f79-vqj2r" Oct 30 17:43:03.262205 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:03.262037 4965 scope.go:122] "RemoveContainer" containerID="4669f588933210f6b5779f12187f150405915dd20395c6f7c38d3e7bb290ba09" Oct 30 17:43:03.268075 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:03.268042 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-143041-6bfddb8867-5tnjk" event={"ID":"f25e38d6-2f5a-4b43-bf0e-630706a343a1","Type":"ContainerStarted","Data":"0390d81e4e710549ea650c97a17a17698796927c245560f83d3a2f1e7bd89dd0"} Oct 30 17:43:03.268186 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:03.268085 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-143041-6bfddb8867-5tnjk" event={"ID":"f25e38d6-2f5a-4b43-bf0e-630706a343a1","Type":"ContainerStarted","Data":"b4d82cad31336923e21e6f7a75ec77e542b8a880ca04dc5f31c6a748b873fcde"} Oct 30 17:43:03.268737 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:03.268718 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-143041-6bfddb8867-5tnjk" Oct 30 17:43:03.277331 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:03.277312 4965 scope.go:122] "RemoveContainer" containerID="4669f588933210f6b5779f12187f150405915dd20395c6f7c38d3e7bb290ba09" Oct 30 17:43:03.277611 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:43:03.277593 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4669f588933210f6b5779f12187f150405915dd20395c6f7c38d3e7bb290ba09\": container with ID starting with 4669f588933210f6b5779f12187f150405915dd20395c6f7c38d3e7bb290ba09 not found: ID does not exist" containerID="4669f588933210f6b5779f12187f150405915dd20395c6f7c38d3e7bb290ba09" Oct 30 17:43:03.277664 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:03.277617 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4669f588933210f6b5779f12187f150405915dd20395c6f7c38d3e7bb290ba09"} err="failed to get container status \"4669f588933210f6b5779f12187f150405915dd20395c6f7c38d3e7bb290ba09\": rpc error: code = NotFound desc = could not find container \"4669f588933210f6b5779f12187f150405915dd20395c6f7c38d3e7bb290ba09\": container with ID starting with 4669f588933210f6b5779f12187f150405915dd20395c6f7c38d3e7bb290ba09 not found: ID does not exist" Oct 30 17:43:03.318179 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:03.318156 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-49e155-5997c8f79-vqj2r"] Oct 30 17:43:03.325885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:03.325848 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-49e155-5997c8f79-vqj2r"] Oct 30 17:43:03.384376 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:03.382110 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-143041-6bfddb8867-5tnjk" podStartSLOduration=1.382087292 podStartE2EDuration="1.382087292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:43:03.376995437 +0000 UTC m=+1841.429095372" watchObservedRunningTime="2026-10-30 17:43:03.382087292 +0000 UTC m=+1841.434187224" Oct 30 17:43:04.100944 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:04.100907 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="112bfc33-9359-4f17-be86-f3ffd6ecae0f" path="/var/lib/kubelet/pods/112bfc33-9359-4f17-be86-f3ffd6ecae0f/volumes" Oct 30 17:43:05.998475 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:43:05.998441 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:43:15.287405 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:15.287371 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-143041-6bfddb8867-5tnjk" Oct 30 17:43:19.841406 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:19.841372 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b1c345-d4898585f-6kvs7"] Oct 30 17:43:19.842280 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:19.842254 4965 cpu_manager.go:414] "containerMap: removing container" podUID="112bfc33-9359-4f17-be86-f3ffd6ecae0f" containerName="payload-pre-processing" Oct 30 17:43:19.842335 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:19.842280 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="112bfc33-9359-4f17-be86-f3ffd6ecae0f" containerName="payload-pre-processing" Oct 30 17:43:19.842405 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:19.842395 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="112bfc33-9359-4f17-be86-f3ffd6ecae0f" containerName="payload-pre-processing" Oct 30 17:43:19.846603 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:19.846584 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1c345-d4898585f-6kvs7" Oct 30 17:43:19.864312 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:19.864295 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b1c345\"" Oct 30 17:43:19.899090 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:19.899059 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b1c345-d4898585f-6kvs7"] Oct 30 17:43:19.957516 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:19.957480 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d95f31-b1ea-4a3c-92a3-1118993629ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b1c345-d4898585f-6kvs7\" (UID: \"e0d95f31-b1ea-4a3c-92a3-1118993629ed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1c345-d4898585f-6kvs7" Oct 30 17:43:19.957993 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:19.957974 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvjhz\" (UniqueName: \"kubernetes.io/projected/e0d95f31-b1ea-4a3c-92a3-1118993629ed-kube-api-access-mvjhz\") pod \"payload-pre-processing-e2e-inf-b-b1c345-d4898585f-6kvs7\" (UID: \"e0d95f31-b1ea-4a3c-92a3-1118993629ed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1c345-d4898585f-6kvs7" Oct 30 17:43:20.301978 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:20.292488 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvjhz\" (UniqueName: \"kubernetes.io/projected/e0d95f31-b1ea-4a3c-92a3-1118993629ed-kube-api-access-mvjhz\") pod \"payload-pre-processing-e2e-inf-b-b1c345-d4898585f-6kvs7\" (UID: \"e0d95f31-b1ea-4a3c-92a3-1118993629ed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1c345-d4898585f-6kvs7" Oct 30 17:43:20.301978 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:20.292643 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d95f31-b1ea-4a3c-92a3-1118993629ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b1c345-d4898585f-6kvs7\" (UID: \"e0d95f31-b1ea-4a3c-92a3-1118993629ed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1c345-d4898585f-6kvs7" Oct 30 17:43:20.312925 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:20.305855 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d95f31-b1ea-4a3c-92a3-1118993629ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b1c345-d4898585f-6kvs7\" (UID: \"e0d95f31-b1ea-4a3c-92a3-1118993629ed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1c345-d4898585f-6kvs7" Oct 30 17:43:20.368260 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:20.368225 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvjhz\" (UniqueName: \"kubernetes.io/projected/e0d95f31-b1ea-4a3c-92a3-1118993629ed-kube-api-access-mvjhz\") pod \"payload-pre-processing-e2e-inf-b-b1c345-d4898585f-6kvs7\" (UID: \"e0d95f31-b1ea-4a3c-92a3-1118993629ed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1c345-d4898585f-6kvs7" Oct 30 17:43:20.462614 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:20.462582 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1c345-d4898585f-6kvs7" Oct 30 17:43:20.770803 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:20.770763 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b1c345-d4898585f-6kvs7"] Oct 30 17:43:20.801935 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:43:20.771305 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d95f31_b1ea_4a3c_92a3_1118993629ed.slice/crio-82bf539e1af0ca58e9b36a8f00f8b17af22bd33358ce67abb8635b9dd82ba0f6 WatchSource:0}: Error finding container 82bf539e1af0ca58e9b36a8f00f8b17af22bd33358ce67abb8635b9dd82ba0f6: Status 404 returned error can't find the container with id 82bf539e1af0ca58e9b36a8f00f8b17af22bd33358ce67abb8635b9dd82ba0f6 Oct 30 17:43:21.476381 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:21.473894 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1c345-d4898585f-6kvs7" event={"ID":"e0d95f31-b1ea-4a3c-92a3-1118993629ed","Type":"ContainerStarted","Data":"a79ea137cf3be37bd33540a40e65140f4dffef8add43a2a423b33a0946e15b5b"} Oct 30 17:43:21.476381 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:21.474008 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1c345-d4898585f-6kvs7" event={"ID":"e0d95f31-b1ea-4a3c-92a3-1118993629ed","Type":"ContainerStarted","Data":"82bf539e1af0ca58e9b36a8f00f8b17af22bd33358ce67abb8635b9dd82ba0f6"} Oct 30 17:43:21.476381 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:21.474228 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1c345-d4898585f-6kvs7" Oct 30 17:43:21.514937 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:21.514881 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1c345-d4898585f-6kvs7" podStartSLOduration=2.514850163 podStartE2EDuration="2.514850163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:43:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:43:21.514181898 +0000 UTC m=+1859.566281821" watchObservedRunningTime="2026-10-30 17:43:21.514850163 +0000 UTC m=+1859.566950093" Oct 30 17:43:22.470231 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:22.470186 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z"] Oct 30 17:43:22.475295 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:22.475270 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:22.495934 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:22.495902 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z"] Oct 30 17:43:22.535730 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:22.535687 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkhv7\" (UniqueName: \"kubernetes.io/projected/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-kube-api-access-mkhv7\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:22.535959 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:22.535886 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istio-podinfo\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:22.535959 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:22.535946 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istiod-ca-cert\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:22.536106 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:22.536001 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-credential-socket\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:22.536248 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:22.536217 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-workload-socket\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:22.536394 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:22.536275 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-workload-certs\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:22.536470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:22.536385 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istio-data\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:22.536470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:22.536457 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istio-token\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:22.536583 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:22.536572 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istio-envoy\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:22.638722 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:22.638671 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkhv7\" (UniqueName: \"kubernetes.io/projected/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-kube-api-access-mkhv7\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:22.638940 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:22.638785 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istio-podinfo\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:22.639024 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:22.638934 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istiod-ca-cert\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:22.639024 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:22.639003 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-credential-socket\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:22.639149 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:22.639129 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-workload-socket\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:22.639239 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:22.639174 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-workload-certs\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:22.639239 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:22.639225 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istio-data\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:22.639352 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:22.639292 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istio-token\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:22.639413 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:22.639385 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istio-envoy\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:22.639943 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:22.639615 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-workload-socket\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:22.639943 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:22.639846 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istiod-ca-cert\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:22.640128 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:22.640028 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-workload-certs\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:22.640371 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:22.640329 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istio-data\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:22.640504 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:22.640475 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-credential-socket\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:22.642036 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:22.642015 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istio-podinfo\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:22.642348 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:22.642331 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istio-envoy\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:22.662328 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:22.662295 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istio-token\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:22.662522 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:22.662503 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkhv7\" (UniqueName: \"kubernetes.io/projected/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-kube-api-access-mkhv7\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:22.797943 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:22.796225 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:23.165297 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:43:23.165269 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74e7c579_cc3b_4bd5_998e_3bc0a50aed85.slice/crio-d635d346c042f300a9dd0e5eab7d279b6e45665a0538e0ed87efcfca5d9ccc39 WatchSource:0}: Error finding container d635d346c042f300a9dd0e5eab7d279b6e45665a0538e0ed87efcfca5d9ccc39: Status 404 returned error can't find the container with id d635d346c042f300a9dd0e5eab7d279b6e45665a0538e0ed87efcfca5d9ccc39 Oct 30 17:43:23.166402 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:23.166380 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z"] Oct 30 17:43:23.168389 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:23.168357 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:43:23.168459 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:23.168419 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:43:23.168459 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:23.168449 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:43:23.486939 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:23.482247 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:43:23.598897 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:23.593852 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" event={"ID":"74e7c579-cc3b-4bd5-998e-3bc0a50aed85","Type":"ContainerStarted","Data":"26c5e7f5f5ca6f56d9e6c998a30f9b49c31a7199766784fc813dda2f5309ed45"} Oct 30 17:43:23.598897 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:23.593939 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" event={"ID":"74e7c579-cc3b-4bd5-998e-3bc0a50aed85","Type":"ContainerStarted","Data":"d635d346c042f300a9dd0e5eab7d279b6e45665a0538e0ed87efcfca5d9ccc39"} Oct 30 17:43:23.660155 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:23.660095 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" podStartSLOduration=1.660073476 podStartE2EDuration="1.660073476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:43:23.65535791 +0000 UTC m=+1861.707457873" watchObservedRunningTime="2026-10-30 17:43:23.660073476 +0000 UTC m=+1861.712173408" Oct 30 17:43:23.797599 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:23.797528 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:24.801895 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:24.801350 4965 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 17:43:24.801895 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:24.801428 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" podUID="74e7c579-cc3b-4bd5-998e-3bc0a50aed85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 17:43:25.797083 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:25.797007 4965 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 17:43:25.797083 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:25.797066 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" podUID="74e7c579-cc3b-4bd5-998e-3bc0a50aed85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 17:43:26.807884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:26.807228 4965 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 17:43:26.807884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:26.807284 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" podUID="74e7c579-cc3b-4bd5-998e-3bc0a50aed85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 17:43:27.803975 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:27.803939 4965 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 17:43:27.804254 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:27.804228 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" podUID="74e7c579-cc3b-4bd5-998e-3bc0a50aed85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 17:43:28.823263 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:28.823226 4965 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 17:43:28.823833 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:28.823800 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" podUID="74e7c579-cc3b-4bd5-998e-3bc0a50aed85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 17:43:29.798343 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:29.798298 4965 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 17:43:29.798674 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:29.798638 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" podUID="74e7c579-cc3b-4bd5-998e-3bc0a50aed85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 17:43:30.846498 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:30.846453 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:30.850682 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:30.848964 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:30.851299 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:30.851237 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:43:32.486150 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:32.486119 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1c345-d4898585f-6kvs7" Oct 30 17:43:34.472781 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:34.472742 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz"] Oct 30 17:43:34.496228 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:34.496181 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" podUID="c7be7daf-3252-4355-99da-2cd6c664b3ae" containerName="istio-proxy" containerID="cri-o://239c0e755645f125174b47c2b8df61144fa464980db515f21a778838e3d567a4" gracePeriod=30 Oct 30 17:43:36.002464 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:43:36.002421 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:43:36.684902 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:36.684842 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh"] Oct 30 17:43:36.691481 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:36.691450 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:36.733731 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:36.733697 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh"] Oct 30 17:43:36.744374 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:36.744337 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd9q9\" (UniqueName: \"kubernetes.io/projected/33457c12-09bd-4268-83a6-3543bd62be67-kube-api-access-gd9q9\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:36.744541 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:36.744385 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-istio-envoy\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:36.744770 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:36.744743 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33457c12-09bd-4268-83a6-3543bd62be67-istiod-ca-cert\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:36.744965 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:36.744846 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-istio-data\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:36.745205 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:36.745127 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-workload-certs\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:36.745205 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:36.745168 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33457c12-09bd-4268-83a6-3543bd62be67-istio-podinfo\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:36.745205 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:36.745197 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-credential-socket\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:36.745396 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:36.745239 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-workload-socket\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:36.745396 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:36.745370 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33457c12-09bd-4268-83a6-3543bd62be67-istio-token\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:36.847130 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:36.847086 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33457c12-09bd-4268-83a6-3543bd62be67-istiod-ca-cert\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:36.847320 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:36.847244 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-istio-data\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:36.847401 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:36.847319 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-workload-certs\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:36.847401 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:36.847345 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33457c12-09bd-4268-83a6-3543bd62be67-istio-podinfo\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:36.847401 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:36.847374 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-credential-socket\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:36.847562 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:36.847420 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-workload-socket\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:36.847624 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:36.847582 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33457c12-09bd-4268-83a6-3543bd62be67-istio-token\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:36.847690 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:36.847611 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-istio-data\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:36.847745 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:36.847707 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd9q9\" (UniqueName: \"kubernetes.io/projected/33457c12-09bd-4268-83a6-3543bd62be67-kube-api-access-gd9q9\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:36.847799 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:36.847737 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-istio-envoy\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:36.847851 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:36.847803 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33457c12-09bd-4268-83a6-3543bd62be67-istiod-ca-cert\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:36.848155 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:36.848129 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-credential-socket\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:36.848260 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:36.848198 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-workload-certs\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:36.848260 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:36.848205 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-workload-socket\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:36.849904 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:36.849880 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-istio-envoy\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:36.850136 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:36.850116 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33457c12-09bd-4268-83a6-3543bd62be67-istio-podinfo\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:36.887494 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:36.887469 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd9q9\" (UniqueName: \"kubernetes.io/projected/33457c12-09bd-4268-83a6-3543bd62be67-kube-api-access-gd9q9\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:36.896249 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:36.888325 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33457c12-09bd-4268-83a6-3543bd62be67-istio-token\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:37.013756 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:37.013732 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:37.341161 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:43:37.341127 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33457c12_09bd_4268_83a6_3543bd62be67.slice/crio-c41adceba9ad8be727be32a0b7b38d9c48c092686f4bcbc511d99e2b0eb85bdb WatchSource:0}: Error finding container c41adceba9ad8be727be32a0b7b38d9c48c092686f4bcbc511d99e2b0eb85bdb: Status 404 returned error can't find the container with id c41adceba9ad8be727be32a0b7b38d9c48c092686f4bcbc511d99e2b0eb85bdb Oct 30 17:43:37.341460 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:37.341437 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh"] Oct 30 17:43:37.344124 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:37.344091 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:43:37.344217 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:37.344159 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:43:37.344217 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:37.344189 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:43:37.814064 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:37.813943 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" event={"ID":"33457c12-09bd-4268-83a6-3543bd62be67","Type":"ContainerStarted","Data":"edb8ac61d2c3c719c9905e8c1632468fdad0efd0aa0daa0a05efb58ce85a3fd0"} Oct 30 17:43:37.814339 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:37.814316 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" event={"ID":"33457c12-09bd-4268-83a6-3543bd62be67","Type":"ContainerStarted","Data":"c41adceba9ad8be727be32a0b7b38d9c48c092686f4bcbc511d99e2b0eb85bdb"} Oct 30 17:43:37.816575 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:37.816541 4965 generic.go:356] "Generic (PLEG): container finished" podID="508ddd74-da86-4674-ad80-7af4a1eb7310" containerID="a00bc22a52abafa1d7649f89563b915129c6562f936fc8ef33227ab8d4aff299" exitCode=0 Oct 30 17:43:37.816719 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:37.816574 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" event={"ID":"508ddd74-da86-4674-ad80-7af4a1eb7310","Type":"ContainerDied","Data":"a00bc22a52abafa1d7649f89563b915129c6562f936fc8ef33227ab8d4aff299"} Oct 30 17:43:37.817011 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:37.816991 4965 scope.go:122] "RemoveContainer" containerID="a00bc22a52abafa1d7649f89563b915129c6562f936fc8ef33227ab8d4aff299" Oct 30 17:43:37.818740 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:37.818713 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:43:37.818816 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:37.818786 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:43:37.818879 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:37.818835 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:43:37.867545 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:37.867484 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" podStartSLOduration=1.8674659409999999 podStartE2EDuration="1.867465941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:43:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:43:37.863537598 +0000 UTC m=+1875.915637668" watchObservedRunningTime="2026-10-30 17:43:37.867465941 +0000 UTC m=+1875.919565931" Oct 30 17:43:38.014206 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:38.014176 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:38.015494 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:38.015469 4965 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 17:43:38.015600 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:38.015512 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" podUID="33457c12-09bd-4268-83a6-3543bd62be67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 17:43:39.016043 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:39.016005 4965 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 17:43:39.016651 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:39.016618 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" podUID="33457c12-09bd-4268-83a6-3543bd62be67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 17:43:39.028887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:39.022653 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" event={"ID":"508ddd74-da86-4674-ad80-7af4a1eb7310","Type":"ContainerStarted","Data":"f294af8eb72ab92728967dc99939fd78b0d055ab55dbe21e483a3559cdc0f587"} Oct 30 17:43:39.888886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:39.887973 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:43:39.921264 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:39.921234 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:43:40.016988 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.010207 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:43:40.018029 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.017985 4965 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 17:43:40.018196 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.018044 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" podUID="33457c12-09bd-4268-83a6-3543bd62be67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 17:43:40.023155 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.023123 4965 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 17:43:40.023291 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.023174 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" podUID="508ddd74-da86-4674-ad80-7af4a1eb7310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 17:43:40.025737 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.025707 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7be7daf-3252-4355-99da-2cd6c664b3ae-istio-token\" (UniqueName: \"kubernetes.io/projected/c7be7daf-3252-4355-99da-2cd6c664b3ae-istio-token\") pod \"c7be7daf-3252-4355-99da-2cd6c664b3ae\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " Oct 30 17:43:40.025895 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.025752 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-istio-data\") pod \"c7be7daf-3252-4355-99da-2cd6c664b3ae\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " Oct 30 17:43:40.025895 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.025845 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-istio-envoy\") pod \"c7be7daf-3252-4355-99da-2cd6c664b3ae\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " Oct 30 17:43:40.026025 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.025913 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-workload-certs\") pod \"c7be7daf-3252-4355-99da-2cd6c664b3ae\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " Oct 30 17:43:40.026025 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.025969 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7be7daf-3252-4355-99da-2cd6c664b3ae-kube-api-access-cmffg\" (UniqueName: \"kubernetes.io/projected/c7be7daf-3252-4355-99da-2cd6c664b3ae-kube-api-access-cmffg\") pod \"c7be7daf-3252-4355-99da-2cd6c664b3ae\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " Oct 30 17:43:40.026143 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.026062 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-workload-socket\") pod \"c7be7daf-3252-4355-99da-2cd6c664b3ae\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " Oct 30 17:43:40.026208 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.026154 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-credential-socket\") pod \"c7be7daf-3252-4355-99da-2cd6c664b3ae\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " Oct 30 17:43:40.026271 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.026205 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7be7daf-3252-4355-99da-2cd6c664b3ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7be7daf-3252-4355-99da-2cd6c664b3ae-istiod-ca-cert\") pod \"c7be7daf-3252-4355-99da-2cd6c664b3ae\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " Oct 30 17:43:40.026493 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.026470 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7be7daf-3252-4355-99da-2cd6c664b3ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7be7daf-3252-4355-99da-2cd6c664b3ae-istio-podinfo\") pod \"c7be7daf-3252-4355-99da-2cd6c664b3ae\" (UID: \"c7be7daf-3252-4355-99da-2cd6c664b3ae\") " Oct 30 17:43:40.026829 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.026804 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-workload-certs" pod "c7be7daf-3252-4355-99da-2cd6c664b3ae" (UID: "c7be7daf-3252-4355-99da-2cd6c664b3ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:43:40.027170 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.027147 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:43:40.027262 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.027204 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-istio-data" pod "c7be7daf-3252-4355-99da-2cd6c664b3ae" (UID: "c7be7daf-3252-4355-99da-2cd6c664b3ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:43:40.027653 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.027416 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-credential-socket" pod "c7be7daf-3252-4355-99da-2cd6c664b3ae" (UID: "c7be7daf-3252-4355-99da-2cd6c664b3ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:43:40.027653 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.027512 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7be7daf-3252-4355-99da-2cd6c664b3ae-istiod-ca-cert" pod "c7be7daf-3252-4355-99da-2cd6c664b3ae" (UID: "c7be7daf-3252-4355-99da-2cd6c664b3ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:43:40.027653 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.027623 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-workload-socket" pod "c7be7daf-3252-4355-99da-2cd6c664b3ae" (UID: "c7be7daf-3252-4355-99da-2cd6c664b3ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:43:40.030551 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.030522 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7be7daf-3252-4355-99da-2cd6c664b3ae-istio-token" pod "c7be7daf-3252-4355-99da-2cd6c664b3ae" (UID: "c7be7daf-3252-4355-99da-2cd6c664b3ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:43:40.030658 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.030635 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-istio-envoy" pod "c7be7daf-3252-4355-99da-2cd6c664b3ae" (UID: "c7be7daf-3252-4355-99da-2cd6c664b3ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:43:40.032350 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.032079 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7be7daf-3252-4355-99da-2cd6c664b3ae-kube-api-access-cmffg" pod "c7be7daf-3252-4355-99da-2cd6c664b3ae" (UID: "c7be7daf-3252-4355-99da-2cd6c664b3ae"). InnerVolumeSpecName "kube-api-access-cmffg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:43:40.032978 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.032764 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7be7daf-3252-4355-99da-2cd6c664b3ae-istio-podinfo" pod "c7be7daf-3252-4355-99da-2cd6c664b3ae" (UID: "c7be7daf-3252-4355-99da-2cd6c664b3ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:43:40.040041 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.040013 4965 generic.go:356] "Generic (PLEG): container finished" podID="c7be7daf-3252-4355-99da-2cd6c664b3ae" containerID="239c0e755645f125174b47c2b8df61144fa464980db515f21a778838e3d567a4" exitCode=0 Oct 30 17:43:40.040151 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.040082 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" Oct 30 17:43:40.040151 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.040095 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" event={"ID":"c7be7daf-3252-4355-99da-2cd6c664b3ae","Type":"ContainerDied","Data":"239c0e755645f125174b47c2b8df61144fa464980db515f21a778838e3d567a4"} Oct 30 17:43:40.040271 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.040156 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz" event={"ID":"c7be7daf-3252-4355-99da-2cd6c664b3ae","Type":"ContainerDied","Data":"b3e76a2eef4a6ee1737284f2b06ac58dd5d95928d377f7d42af43825b8782002"} Oct 30 17:43:40.040271 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.040191 4965 scope.go:122] "RemoveContainer" containerID="239c0e755645f125174b47c2b8df61144fa464980db515f21a778838e3d567a4" Oct 30 17:43:40.123957 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.123940 4965 scope.go:122] "RemoveContainer" containerID="408a5544a685adbd16ef188d221b11569357182c675be37b22f76d5be83094ca" Oct 30 17:43:40.128273 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.128249 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7be7daf-3252-4355-99da-2cd6c664b3ae-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:43:40.128273 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.128273 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7be7daf-3252-4355-99da-2cd6c664b3ae-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:43:40.128412 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.128283 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:43:40.128412 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.128292 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:43:40.128412 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.128302 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmffg\" (UniqueName: \"kubernetes.io/projected/c7be7daf-3252-4355-99da-2cd6c664b3ae-kube-api-access-cmffg\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:43:40.128412 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.128310 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:43:40.128412 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.128320 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7be7daf-3252-4355-99da-2cd6c664b3ae-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:43:40.128412 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.128328 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7be7daf-3252-4355-99da-2cd6c664b3ae-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:43:40.136090 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.136072 4965 scope.go:122] "RemoveContainer" containerID="239c0e755645f125174b47c2b8df61144fa464980db515f21a778838e3d567a4" Oct 30 17:43:40.136412 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:43:40.136385 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"239c0e755645f125174b47c2b8df61144fa464980db515f21a778838e3d567a4\": container with ID starting with 239c0e755645f125174b47c2b8df61144fa464980db515f21a778838e3d567a4 not found: ID does not exist" containerID="239c0e755645f125174b47c2b8df61144fa464980db515f21a778838e3d567a4" Oct 30 17:43:40.136499 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.136425 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"239c0e755645f125174b47c2b8df61144fa464980db515f21a778838e3d567a4"} err="failed to get container status \"239c0e755645f125174b47c2b8df61144fa464980db515f21a778838e3d567a4\": rpc error: code = NotFound desc = could not find container \"239c0e755645f125174b47c2b8df61144fa464980db515f21a778838e3d567a4\": container with ID starting with 239c0e755645f125174b47c2b8df61144fa464980db515f21a778838e3d567a4 not found: ID does not exist" Oct 30 17:43:40.136499 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.136449 4965 scope.go:122] "RemoveContainer" containerID="408a5544a685adbd16ef188d221b11569357182c675be37b22f76d5be83094ca" Oct 30 17:43:40.136745 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:43:40.136729 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"408a5544a685adbd16ef188d221b11569357182c675be37b22f76d5be83094ca\": container with ID starting with 408a5544a685adbd16ef188d221b11569357182c675be37b22f76d5be83094ca not found: ID does not exist" containerID="408a5544a685adbd16ef188d221b11569357182c675be37b22f76d5be83094ca" Oct 30 17:43:40.136810 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.136751 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"408a5544a685adbd16ef188d221b11569357182c675be37b22f76d5be83094ca"} err="failed to get container status \"408a5544a685adbd16ef188d221b11569357182c675be37b22f76d5be83094ca\": rpc error: code = NotFound desc = could not find container \"408a5544a685adbd16ef188d221b11569357182c675be37b22f76d5be83094ca\": container with ID starting with 408a5544a685adbd16ef188d221b11569357182c675be37b22f76d5be83094ca not found: ID does not exist" Oct 30 17:43:40.211989 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.211943 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz"] Oct 30 17:43:40.215919 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.215893 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-802bda-openshift-default-664b986cf5-689jz"] Oct 30 17:43:40.962954 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.962915 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-143041-5cf495455f-fcxtn"] Oct 30 17:43:40.979697 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.979650 4965 cpu_manager.go:414] "containerMap: removing container" podUID="c7be7daf-3252-4355-99da-2cd6c664b3ae" containerName="istio-proxy" Oct 30 17:43:40.979955 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.979929 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7be7daf-3252-4355-99da-2cd6c664b3ae" containerName="istio-proxy" Oct 30 17:43:40.980252 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.980233 4965 cpu_manager.go:414] "containerMap: removing container" podUID="c7be7daf-3252-4355-99da-2cd6c664b3ae" containerName="istio-proxy" Oct 30 17:43:40.980375 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.980362 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7be7daf-3252-4355-99da-2cd6c664b3ae" containerName="istio-proxy" Oct 30 17:43:40.980697 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.980678 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7be7daf-3252-4355-99da-2cd6c664b3ae" containerName="istio-proxy" Oct 30 17:43:40.980836 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:40.980824 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7be7daf-3252-4355-99da-2cd6c664b3ae" containerName="istio-proxy" Oct 30 17:43:41.025889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:41.017280 4965 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 17:43:41.025889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:41.017347 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" podUID="508ddd74-da86-4674-ad80-7af4a1eb7310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 17:43:41.062880 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:41.018687 4965 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 17:43:41.062880 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:41.062509 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" podUID="33457c12-09bd-4268-83a6-3543bd62be67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 17:43:41.448841 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:41.448807 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-143041-5cf495455f-fcxtn"] Oct 30 17:43:41.449285 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:41.449187 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-143041-5cf495455f-fcxtn" Oct 30 17:43:41.511927 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:41.482545 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-143041-serving-cert\"" Oct 30 17:43:41.625261 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:41.625220 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51bd6cbd-5631-41f9-9dd8-be43832c6d32-openshift-service-ca\") pod \"maas-api-e2e-inf-b-143041-5cf495455f-fcxtn\" (UID: \"51bd6cbd-5631-41f9-9dd8-be43832c6d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-143041-5cf495455f-fcxtn" Oct 30 17:43:41.625490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:41.625467 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxdzk\" (UniqueName: \"kubernetes.io/projected/51bd6cbd-5631-41f9-9dd8-be43832c6d32-kube-api-access-nxdzk\") pod \"maas-api-e2e-inf-b-143041-5cf495455f-fcxtn\" (UID: \"51bd6cbd-5631-41f9-9dd8-be43832c6d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-143041-5cf495455f-fcxtn" Oct 30 17:43:41.625580 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:41.625546 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51bd6cbd-5631-41f9-9dd8-be43832c6d32-metrics-certs\") pod \"maas-api-e2e-inf-b-143041-5cf495455f-fcxtn\" (UID: \"51bd6cbd-5631-41f9-9dd8-be43832c6d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-143041-5cf495455f-fcxtn" Oct 30 17:43:41.625654 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:41.625639 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51bd6cbd-5631-41f9-9dd8-be43832c6d32-maas-api-tls\") pod \"maas-api-e2e-inf-b-143041-5cf495455f-fcxtn\" (UID: \"51bd6cbd-5631-41f9-9dd8-be43832c6d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-143041-5cf495455f-fcxtn" Oct 30 17:43:41.727224 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:41.727121 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxdzk\" (UniqueName: \"kubernetes.io/projected/51bd6cbd-5631-41f9-9dd8-be43832c6d32-kube-api-access-nxdzk\") pod \"maas-api-e2e-inf-b-143041-5cf495455f-fcxtn\" (UID: \"51bd6cbd-5631-41f9-9dd8-be43832c6d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-143041-5cf495455f-fcxtn" Oct 30 17:43:41.727419 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:41.727229 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51bd6cbd-5631-41f9-9dd8-be43832c6d32-metrics-certs\") pod \"maas-api-e2e-inf-b-143041-5cf495455f-fcxtn\" (UID: \"51bd6cbd-5631-41f9-9dd8-be43832c6d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-143041-5cf495455f-fcxtn" Oct 30 17:43:41.727419 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:41.727308 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51bd6cbd-5631-41f9-9dd8-be43832c6d32-maas-api-tls\") pod \"maas-api-e2e-inf-b-143041-5cf495455f-fcxtn\" (UID: \"51bd6cbd-5631-41f9-9dd8-be43832c6d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-143041-5cf495455f-fcxtn" Oct 30 17:43:41.727419 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:41.727409 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51bd6cbd-5631-41f9-9dd8-be43832c6d32-openshift-service-ca\") pod \"maas-api-e2e-inf-b-143041-5cf495455f-fcxtn\" (UID: \"51bd6cbd-5631-41f9-9dd8-be43832c6d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-143041-5cf495455f-fcxtn" Oct 30 17:43:41.728167 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:41.728143 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51bd6cbd-5631-41f9-9dd8-be43832c6d32-openshift-service-ca\") pod \"maas-api-e2e-inf-b-143041-5cf495455f-fcxtn\" (UID: \"51bd6cbd-5631-41f9-9dd8-be43832c6d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-143041-5cf495455f-fcxtn" Oct 30 17:43:41.769558 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:41.756588 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51bd6cbd-5631-41f9-9dd8-be43832c6d32-maas-api-tls\") pod \"maas-api-e2e-inf-b-143041-5cf495455f-fcxtn\" (UID: \"51bd6cbd-5631-41f9-9dd8-be43832c6d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-143041-5cf495455f-fcxtn" Oct 30 17:43:41.770400 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:41.770371 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51bd6cbd-5631-41f9-9dd8-be43832c6d32-metrics-certs\") pod \"maas-api-e2e-inf-b-143041-5cf495455f-fcxtn\" (UID: \"51bd6cbd-5631-41f9-9dd8-be43832c6d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-143041-5cf495455f-fcxtn" Oct 30 17:43:41.811963 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:41.811637 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxdzk\" (UniqueName: \"kubernetes.io/projected/51bd6cbd-5631-41f9-9dd8-be43832c6d32-kube-api-access-nxdzk\") pod \"maas-api-e2e-inf-b-143041-5cf495455f-fcxtn\" (UID: \"51bd6cbd-5631-41f9-9dd8-be43832c6d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-143041-5cf495455f-fcxtn" Oct 30 17:43:41.902917 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:41.902477 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-143041-5cf495455f-fcxtn" Oct 30 17:43:42.017946 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:42.017803 4965 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 17:43:42.017946 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:42.017892 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" podUID="508ddd74-da86-4674-ad80-7af4a1eb7310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 17:43:42.022873 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:42.022826 4965 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 17:43:42.023015 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:42.022901 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" podUID="33457c12-09bd-4268-83a6-3543bd62be67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 17:43:42.128883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:42.128076 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7be7daf-3252-4355-99da-2cd6c664b3ae" path="/var/lib/kubelet/pods/c7be7daf-3252-4355-99da-2cd6c664b3ae/volumes" Oct 30 17:43:42.373291 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:42.373259 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-143041-5cf495455f-fcxtn"] Oct 30 17:43:43.011745 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:43.011710 4965 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 17:43:43.012076 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:43.012045 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" podUID="508ddd74-da86-4674-ad80-7af4a1eb7310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 17:43:43.017300 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:43.016190 4965 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 17:43:43.017300 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:43.016242 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" podUID="33457c12-09bd-4268-83a6-3543bd62be67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 17:43:43.552929 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:43.552468 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-143041-5cf495455f-fcxtn" event={"ID":"51bd6cbd-5631-41f9-9dd8-be43832c6d32","Type":"ContainerStarted","Data":"7797d9af74c45352f287c4cabf796d04b6cf0ac11896d384759faae8014991f3"} Oct 30 17:43:43.552929 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:43.552542 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-143041-5cf495455f-fcxtn" event={"ID":"51bd6cbd-5631-41f9-9dd8-be43832c6d32","Type":"ContainerStarted","Data":"4bd99483cda9e5c8cc4da62e9005044b202477f8478603f2811e712d680522bb"} Oct 30 17:43:43.556954 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:43.553775 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-143041-5cf495455f-fcxtn" Oct 30 17:43:43.630890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:43.624220 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-143041-5cf495455f-fcxtn" podStartSLOduration=3.182450008 podStartE2EDuration="3.62419765s" totalImagesPullingTime="441.747642ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:43:43.623473748 +0000 UTC m=+1881.675573680" watchObservedRunningTime="2026-10-30 17:43:43.62419765 +0000 UTC m=+1881.676297628" Oct 30 17:43:44.015886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:44.015259 4965 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 17:43:44.015886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:44.015321 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" podUID="508ddd74-da86-4674-ad80-7af4a1eb7310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 17:43:44.016176 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:44.015947 4965 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 17:43:44.016176 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:44.015987 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" podUID="33457c12-09bd-4268-83a6-3543bd62be67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 17:43:45.010269 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:45.010220 4965 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 17:43:45.010800 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:45.010303 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" podUID="508ddd74-da86-4674-ad80-7af4a1eb7310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 17:43:45.014129 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:45.014101 4965 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 17:43:45.014271 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:45.014157 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" podUID="33457c12-09bd-4268-83a6-3543bd62be67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 17:43:45.411825 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:45.411782 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ee9ff3-55d6b46fd7-rvtsx"] Oct 30 17:43:45.416659 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:45.416634 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee9ff3-55d6b46fd7-rvtsx" Oct 30 17:43:45.477147 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:45.477108 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ee9ff3-dockercfg-2th2j\"" Oct 30 17:43:45.477350 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:45.477238 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ee9ff3\"" Oct 30 17:43:45.591886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:45.584441 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d5970a7-8467-4d2f-99ad-c9f6cc55df7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ee9ff3-55d6b46fd7-rvtsx\" (UID: \"6d5970a7-8467-4d2f-99ad-c9f6cc55df7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee9ff3-55d6b46fd7-rvtsx" Oct 30 17:43:45.591886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:45.584546 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4p68\" (UniqueName: \"kubernetes.io/projected/6d5970a7-8467-4d2f-99ad-c9f6cc55df7d-kube-api-access-k4p68\") pod \"payload-pre-processing-e2e-inf-b-ee9ff3-55d6b46fd7-rvtsx\" (UID: \"6d5970a7-8467-4d2f-99ad-c9f6cc55df7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee9ff3-55d6b46fd7-rvtsx" Oct 30 17:43:45.610898 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:45.608193 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ee9ff3-55d6b46fd7-rvtsx"] Oct 30 17:43:45.766946 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:45.761185 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d5970a7-8467-4d2f-99ad-c9f6cc55df7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ee9ff3-55d6b46fd7-rvtsx\" (UID: \"6d5970a7-8467-4d2f-99ad-c9f6cc55df7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee9ff3-55d6b46fd7-rvtsx" Oct 30 17:43:45.766946 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:45.761319 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4p68\" (UniqueName: \"kubernetes.io/projected/6d5970a7-8467-4d2f-99ad-c9f6cc55df7d-kube-api-access-k4p68\") pod \"payload-pre-processing-e2e-inf-b-ee9ff3-55d6b46fd7-rvtsx\" (UID: \"6d5970a7-8467-4d2f-99ad-c9f6cc55df7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee9ff3-55d6b46fd7-rvtsx" Oct 30 17:43:45.829056 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:45.808049 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d5970a7-8467-4d2f-99ad-c9f6cc55df7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ee9ff3-55d6b46fd7-rvtsx\" (UID: \"6d5970a7-8467-4d2f-99ad-c9f6cc55df7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee9ff3-55d6b46fd7-rvtsx" Oct 30 17:43:45.921882 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:45.881042 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4p68\" (UniqueName: \"kubernetes.io/projected/6d5970a7-8467-4d2f-99ad-c9f6cc55df7d-kube-api-access-k4p68\") pod \"payload-pre-processing-e2e-inf-b-ee9ff3-55d6b46fd7-rvtsx\" (UID: \"6d5970a7-8467-4d2f-99ad-c9f6cc55df7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee9ff3-55d6b46fd7-rvtsx" Oct 30 17:43:46.051524 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:46.051388 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee9ff3-55d6b46fd7-rvtsx" Oct 30 17:43:46.055135 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:46.055101 4965 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 17:43:46.055347 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:46.055320 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" podUID="508ddd74-da86-4674-ad80-7af4a1eb7310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 17:43:46.249538 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:46.248566 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:46.249751 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:46.249718 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:46.253226 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:46.253165 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:43:46.481909 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:46.481878 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ee9ff3-55d6b46fd7-rvtsx"] Oct 30 17:43:46.610287 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:46.610248 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cc4e96-758d6bddb4-zvx88"] Oct 30 17:43:46.789427 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:46.789335 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee9ff3-55d6b46fd7-rvtsx" event={"ID":"6d5970a7-8467-4d2f-99ad-c9f6cc55df7d","Type":"ContainerStarted","Data":"fabcbb54022e689537bd0c5d340bda45071abbc6262683c4667194e1edae49d4"} Oct 30 17:43:46.789427 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:46.789393 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee9ff3-55d6b46fd7-rvtsx" event={"ID":"6d5970a7-8467-4d2f-99ad-c9f6cc55df7d","Type":"ContainerStarted","Data":"1c4086a117c37af008cdcb13f7c4957fcfea709446ade4eb9fd2ac40c771c82a"} Oct 30 17:43:46.789660 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:46.789480 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee9ff3-55d6b46fd7-rvtsx" Oct 30 17:43:46.835777 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:46.835730 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee9ff3-55d6b46fd7-rvtsx" podStartSLOduration=1.835718102 podStartE2EDuration="1.835718102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:43:46.834273435 +0000 UTC m=+1884.886373343" watchObservedRunningTime="2026-10-30 17:43:46.835718102 +0000 UTC m=+1884.887818066" Oct 30 17:43:47.052404 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:47.052317 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:43:47.080735 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:47.080702 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:43:50.488702 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:50.488654 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt"] Oct 30 17:43:50.495621 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:50.495599 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt" Oct 30 17:43:50.507022 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:50.506995 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt"] Oct 30 17:43:50.513524 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:50.513502 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-24fb0d-dockercfg-ssh97\"" Oct 30 17:43:50.548454 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:50.548409 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-24fb0d\"" Oct 30 17:43:50.750374 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:50.750277 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57ba78c3-6a27-4669-8182-35287812b3a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt\" (UID: \"57ba78c3-6a27-4669-8182-35287812b3a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt" Oct 30 17:43:50.858031 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:50.857998 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkj6b\" (UniqueName: \"kubernetes.io/projected/57ba78c3-6a27-4669-8182-35287812b3a1-kube-api-access-dkj6b\") pod \"payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt\" (UID: \"57ba78c3-6a27-4669-8182-35287812b3a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt" Oct 30 17:43:50.871131 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:50.871093 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-143041-5cf495455f-fcxtn" Oct 30 17:43:50.963777 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:50.963742 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57ba78c3-6a27-4669-8182-35287812b3a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt\" (UID: \"57ba78c3-6a27-4669-8182-35287812b3a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt" Oct 30 17:43:50.964006 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:50.963799 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkj6b\" (UniqueName: \"kubernetes.io/projected/57ba78c3-6a27-4669-8182-35287812b3a1-kube-api-access-dkj6b\") pod \"payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt\" (UID: \"57ba78c3-6a27-4669-8182-35287812b3a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt" Oct 30 17:43:50.964703 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:50.964677 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57ba78c3-6a27-4669-8182-35287812b3a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt\" (UID: \"57ba78c3-6a27-4669-8182-35287812b3a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt" Oct 30 17:43:50.989014 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:50.988982 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkj6b\" (UniqueName: \"kubernetes.io/projected/57ba78c3-6a27-4669-8182-35287812b3a1-kube-api-access-dkj6b\") pod \"payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt\" (UID: \"57ba78c3-6a27-4669-8182-35287812b3a1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt" Oct 30 17:43:51.113924 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:51.113898 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt" Oct 30 17:43:51.470654 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:51.470623 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt"] Oct 30 17:43:51.471034 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:43:51.471006 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ba78c3_6a27_4669_8182_35287812b3a1.slice/crio-49ff056d674833ee9798fb00f1123c130e98fa4fc992380dd84efdeeb1adfad2 WatchSource:0}: Error finding container 49ff056d674833ee9798fb00f1123c130e98fa4fc992380dd84efdeeb1adfad2: Status 404 returned error can't find the container with id 49ff056d674833ee9798fb00f1123c130e98fa4fc992380dd84efdeeb1adfad2 Oct 30 17:43:51.610997 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:51.610960 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-cc4e96-758d6bddb4-zvx88" podUID="7be33b0f-8e47-4fcc-a091-8aaec48cd6bd" containerName="payload-processing" containerID="cri-o://fae2241323d75bd4f5e018be322fce01bcda07785f89d6d10a9d0cd7e77e1365" gracePeriod=25 Oct 30 17:43:51.867290 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:51.867264 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cc4e96-758d6bddb4-zvx88" Oct 30 17:43:51.886183 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:51.886150 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7be33b0f-8e47-4fcc-a091-8aaec48cd6bd-kube-api-access-jh6l9\" (UniqueName: \"kubernetes.io/projected/7be33b0f-8e47-4fcc-a091-8aaec48cd6bd-kube-api-access-jh6l9\") pod \"7be33b0f-8e47-4fcc-a091-8aaec48cd6bd\" (UID: \"7be33b0f-8e47-4fcc-a091-8aaec48cd6bd\") " Oct 30 17:43:51.886355 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:51.886340 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7be33b0f-8e47-4fcc-a091-8aaec48cd6bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7be33b0f-8e47-4fcc-a091-8aaec48cd6bd-plugins-config-volume\") pod \"7be33b0f-8e47-4fcc-a091-8aaec48cd6bd\" (UID: \"7be33b0f-8e47-4fcc-a091-8aaec48cd6bd\") " Oct 30 17:43:51.886711 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:51.886685 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7be33b0f-8e47-4fcc-a091-8aaec48cd6bd-plugins-config-volume" pod "7be33b0f-8e47-4fcc-a091-8aaec48cd6bd" (UID: "7be33b0f-8e47-4fcc-a091-8aaec48cd6bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:43:51.886980 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:51.886965 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7be33b0f-8e47-4fcc-a091-8aaec48cd6bd-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:43:51.888234 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:51.888216 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7be33b0f-8e47-4fcc-a091-8aaec48cd6bd-kube-api-access-jh6l9" pod "7be33b0f-8e47-4fcc-a091-8aaec48cd6bd" (UID: "7be33b0f-8e47-4fcc-a091-8aaec48cd6bd"). InnerVolumeSpecName "kube-api-access-jh6l9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:43:51.959049 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:51.959011 4965 generic.go:356] "Generic (PLEG): container finished" podID="7be33b0f-8e47-4fcc-a091-8aaec48cd6bd" containerID="fae2241323d75bd4f5e018be322fce01bcda07785f89d6d10a9d0cd7e77e1365" exitCode=0 Oct 30 17:43:51.959206 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:51.959057 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cc4e96-758d6bddb4-zvx88" Oct 30 17:43:51.959206 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:51.959085 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cc4e96-758d6bddb4-zvx88" event={"ID":"7be33b0f-8e47-4fcc-a091-8aaec48cd6bd","Type":"ContainerDied","Data":"fae2241323d75bd4f5e018be322fce01bcda07785f89d6d10a9d0cd7e77e1365"} Oct 30 17:43:51.959206 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:51.959142 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cc4e96-758d6bddb4-zvx88" event={"ID":"7be33b0f-8e47-4fcc-a091-8aaec48cd6bd","Type":"ContainerDied","Data":"4dffb43a223f1bb89a95d4dc6617316859d503f7823cee6c836eeea68a827bf7"} Oct 30 17:43:51.959206 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:51.959164 4965 scope.go:122] "RemoveContainer" containerID="fae2241323d75bd4f5e018be322fce01bcda07785f89d6d10a9d0cd7e77e1365" Oct 30 17:43:51.961270 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:51.961239 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt" event={"ID":"57ba78c3-6a27-4669-8182-35287812b3a1","Type":"ContainerStarted","Data":"31132fa8b235d7674e9303750a32e2d9a02c2611b55d17fa1f6d64d9ee464bf3"} Oct 30 17:43:51.961391 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:51.961307 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt" event={"ID":"57ba78c3-6a27-4669-8182-35287812b3a1","Type":"ContainerStarted","Data":"49ff056d674833ee9798fb00f1123c130e98fa4fc992380dd84efdeeb1adfad2"} Oct 30 17:43:51.961391 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:51.961341 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt" Oct 30 17:43:51.970898 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:51.970852 4965 scope.go:122] "RemoveContainer" containerID="fae2241323d75bd4f5e018be322fce01bcda07785f89d6d10a9d0cd7e77e1365" Oct 30 17:43:51.971176 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:43:51.971157 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fae2241323d75bd4f5e018be322fce01bcda07785f89d6d10a9d0cd7e77e1365\": container with ID starting with fae2241323d75bd4f5e018be322fce01bcda07785f89d6d10a9d0cd7e77e1365 not found: ID does not exist" containerID="fae2241323d75bd4f5e018be322fce01bcda07785f89d6d10a9d0cd7e77e1365" Oct 30 17:43:51.971220 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:51.971183 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fae2241323d75bd4f5e018be322fce01bcda07785f89d6d10a9d0cd7e77e1365"} err="failed to get container status \"fae2241323d75bd4f5e018be322fce01bcda07785f89d6d10a9d0cd7e77e1365\": rpc error: code = NotFound desc = could not find container \"fae2241323d75bd4f5e018be322fce01bcda07785f89d6d10a9d0cd7e77e1365\": container with ID starting with fae2241323d75bd4f5e018be322fce01bcda07785f89d6d10a9d0cd7e77e1365 not found: ID does not exist" Oct 30 17:43:51.988942 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:51.988886 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh6l9\" (UniqueName: \"kubernetes.io/projected/7be33b0f-8e47-4fcc-a091-8aaec48cd6bd-kube-api-access-jh6l9\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:43:52.016927 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:52.016848 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cc4e96-758d6bddb4-zvx88"] Oct 30 17:43:52.021351 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:52.021321 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cc4e96-758d6bddb4-zvx88"] Oct 30 17:43:52.123534 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:52.119242 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt" podStartSLOduration=2.119217845 podStartE2EDuration="2.119217845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:43:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:43:52.055144297 +0000 UTC m=+1890.107244246" watchObservedRunningTime="2026-10-30 17:43:52.119217845 +0000 UTC m=+1890.171317776" Oct 30 17:43:52.189132 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:52.189099 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7be33b0f-8e47-4fcc-a091-8aaec48cd6bd" path="/var/lib/kubelet/pods/7be33b0f-8e47-4fcc-a091-8aaec48cd6bd/volumes" Oct 30 17:43:57.801780 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:43:57.801752 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee9ff3-55d6b46fd7-rvtsx" Oct 30 17:44:02.979484 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:02.979449 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt" Oct 30 17:44:05.999434 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:44:05.999398 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:44:09.837507 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:09.837469 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:44:25.554035 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:25.553995 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg"] Oct 30 17:44:25.554660 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:25.554235 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" podUID="9157de31-170a-411a-9046-82c4d4f19231" containerName="istio-proxy" containerID="cri-o://a60c2596d9f7ba2f106503fcc14b1ebfa0b55eabd53cd362f829ca0a02e613ab" gracePeriod=30 Oct 30 17:44:25.560207 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:25.560182 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp"] Oct 30 17:44:25.560478 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:25.560432 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" podUID="5e4b4d14-b581-4844-a80e-a0edd473599e" containerName="istio-proxy" containerID="cri-o://0a4a72091f5c196bcdd521bfd19c5a7cdae4adbec5da81bc8389c2f65fda8f72" gracePeriod=30 Oct 30 17:44:25.990690 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:25.990657 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p"] Oct 30 17:44:25.990946 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:25.990915 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" podUID="5e25ba26-3a38-490c-bc1a-199cfdc132a9" containerName="istio-proxy" containerID="cri-o://2cc49f3d03304661d4f2f2ba290b890c835ef30e7ec4d107a5de327d435b1bc3" gracePeriod=30 Oct 30 17:44:29.640553 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:29.640513 4965 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:44:29.640980 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:29.640588 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" podUID="9157de31-170a-411a-9046-82c4d4f19231" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:44:30.739371 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:30.739332 4965 generic.go:356] "Generic (PLEG): container finished" podID="9157de31-170a-411a-9046-82c4d4f19231" containerID="a60c2596d9f7ba2f106503fcc14b1ebfa0b55eabd53cd362f829ca0a02e613ab" exitCode=0 Oct 30 17:44:30.739929 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:30.739377 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" event={"ID":"9157de31-170a-411a-9046-82c4d4f19231","Type":"ContainerDied","Data":"a60c2596d9f7ba2f106503fcc14b1ebfa0b55eabd53cd362f829ca0a02e613ab"} Oct 30 17:44:30.742307 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:30.742282 4965 generic.go:356] "Generic (PLEG): container finished" podID="5e4b4d14-b581-4844-a80e-a0edd473599e" containerID="0a4a72091f5c196bcdd521bfd19c5a7cdae4adbec5da81bc8389c2f65fda8f72" exitCode=0 Oct 30 17:44:30.742437 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:30.742351 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" event={"ID":"5e4b4d14-b581-4844-a80e-a0edd473599e","Type":"ContainerDied","Data":"0a4a72091f5c196bcdd521bfd19c5a7cdae4adbec5da81bc8389c2f65fda8f72"} Oct 30 17:44:30.921447 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:30.921077 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:44:30.963248 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:30.963222 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:44:31.017491 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.017347 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-workload-socket\") pod \"5e4b4d14-b581-4844-a80e-a0edd473599e\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " Oct 30 17:44:31.017491 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.017468 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e4b4d14-b581-4844-a80e-a0edd473599e-kube-api-access-hrrpv\" (UniqueName: \"kubernetes.io/projected/5e4b4d14-b581-4844-a80e-a0edd473599e-kube-api-access-hrrpv\") pod \"5e4b4d14-b581-4844-a80e-a0edd473599e\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " Oct 30 17:44:31.021795 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.018036 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-istio-envoy\") pod \"5e4b4d14-b581-4844-a80e-a0edd473599e\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " Oct 30 17:44:31.021795 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.018097 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-istio-envoy\") pod \"9157de31-170a-411a-9046-82c4d4f19231\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " Oct 30 17:44:31.021795 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.018132 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-workload-certs\") pod \"9157de31-170a-411a-9046-82c4d4f19231\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " Oct 30 17:44:31.021795 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.018159 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-istio-data\") pod \"5e4b4d14-b581-4844-a80e-a0edd473599e\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " Oct 30 17:44:31.021795 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.018207 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-credential-socket\") pod \"5e4b4d14-b581-4844-a80e-a0edd473599e\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " Oct 30 17:44:31.021795 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.018238 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9157de31-170a-411a-9046-82c4d4f19231-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9157de31-170a-411a-9046-82c4d4f19231-istiod-ca-cert\") pod \"9157de31-170a-411a-9046-82c4d4f19231\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " Oct 30 17:44:31.021795 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.018329 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9157de31-170a-411a-9046-82c4d4f19231-istio-token\" (UniqueName: \"kubernetes.io/projected/9157de31-170a-411a-9046-82c4d4f19231-istio-token\") pod \"9157de31-170a-411a-9046-82c4d4f19231\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " Oct 30 17:44:31.021795 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.018384 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-istio-data\") pod \"9157de31-170a-411a-9046-82c4d4f19231\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " Oct 30 17:44:31.021795 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.018415 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-credential-socket\") pod \"9157de31-170a-411a-9046-82c4d4f19231\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " Oct 30 17:44:31.021795 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.018514 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e4b4d14-b581-4844-a80e-a0edd473599e-istio-token\" (UniqueName: \"kubernetes.io/projected/5e4b4d14-b581-4844-a80e-a0edd473599e-istio-token\") pod \"5e4b4d14-b581-4844-a80e-a0edd473599e\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " Oct 30 17:44:31.021795 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.018567 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-workload-socket\") pod \"9157de31-170a-411a-9046-82c4d4f19231\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " Oct 30 17:44:31.021795 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.018651 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-workload-certs\") pod \"5e4b4d14-b581-4844-a80e-a0edd473599e\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " Oct 30 17:44:31.021795 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.018701 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e4b4d14-b581-4844-a80e-a0edd473599e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e4b4d14-b581-4844-a80e-a0edd473599e-istiod-ca-cert\") pod \"5e4b4d14-b581-4844-a80e-a0edd473599e\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " Oct 30 17:44:31.021795 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.018785 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9157de31-170a-411a-9046-82c4d4f19231-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9157de31-170a-411a-9046-82c4d4f19231-istio-podinfo\") pod \"9157de31-170a-411a-9046-82c4d4f19231\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " Oct 30 17:44:31.021795 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.018826 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e4b4d14-b581-4844-a80e-a0edd473599e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e4b4d14-b581-4844-a80e-a0edd473599e-istio-podinfo\") pod \"5e4b4d14-b581-4844-a80e-a0edd473599e\" (UID: \"5e4b4d14-b581-4844-a80e-a0edd473599e\") " Oct 30 17:44:31.021795 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.018971 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9157de31-170a-411a-9046-82c4d4f19231-kube-api-access-pxtk5\" (UniqueName: \"kubernetes.io/projected/9157de31-170a-411a-9046-82c4d4f19231-kube-api-access-pxtk5\") pod \"9157de31-170a-411a-9046-82c4d4f19231\" (UID: \"9157de31-170a-411a-9046-82c4d4f19231\") " Oct 30 17:44:31.021795 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.018203 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-workload-socket" pod "5e4b4d14-b581-4844-a80e-a0edd473599e" (UID: "5e4b4d14-b581-4844-a80e-a0edd473599e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:31.021795 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.019051 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-workload-socket" pod "9157de31-170a-411a-9046-82c4d4f19231" (UID: "9157de31-170a-411a-9046-82c4d4f19231"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:31.021795 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.019205 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-istio-data" pod "9157de31-170a-411a-9046-82c4d4f19231" (UID: "9157de31-170a-411a-9046-82c4d4f19231"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:31.021795 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.019310 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-credential-socket" pod "9157de31-170a-411a-9046-82c4d4f19231" (UID: "9157de31-170a-411a-9046-82c4d4f19231"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:31.021795 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.019521 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9157de31-170a-411a-9046-82c4d4f19231-istiod-ca-cert" pod "9157de31-170a-411a-9046-82c4d4f19231" (UID: "9157de31-170a-411a-9046-82c4d4f19231"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:44:31.021795 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.021240 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-workload-certs" pod "9157de31-170a-411a-9046-82c4d4f19231" (UID: "9157de31-170a-411a-9046-82c4d4f19231"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:31.031297 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.031129 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-istio-data" pod "5e4b4d14-b581-4844-a80e-a0edd473599e" (UID: "5e4b4d14-b581-4844-a80e-a0edd473599e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:31.031297 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.031190 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-workload-certs" pod "5e4b4d14-b581-4844-a80e-a0edd473599e" (UID: "5e4b4d14-b581-4844-a80e-a0edd473599e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:31.031461 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.031431 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e4b4d14-b581-4844-a80e-a0edd473599e-istiod-ca-cert" pod "5e4b4d14-b581-4844-a80e-a0edd473599e" (UID: "5e4b4d14-b581-4844-a80e-a0edd473599e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:44:31.034941 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.034912 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-credential-socket" pod "5e4b4d14-b581-4844-a80e-a0edd473599e" (UID: "5e4b4d14-b581-4844-a80e-a0edd473599e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:31.035402 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.035375 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e4b4d14-b581-4844-a80e-a0edd473599e-istio-podinfo" pod "5e4b4d14-b581-4844-a80e-a0edd473599e" (UID: "5e4b4d14-b581-4844-a80e-a0edd473599e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:44:31.037959 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.037924 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e4b4d14-b581-4844-a80e-a0edd473599e-istio-token" pod "5e4b4d14-b581-4844-a80e-a0edd473599e" (UID: "5e4b4d14-b581-4844-a80e-a0edd473599e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:44:31.039277 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.039251 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9157de31-170a-411a-9046-82c4d4f19231-kube-api-access-pxtk5" pod "9157de31-170a-411a-9046-82c4d4f19231" (UID: "9157de31-170a-411a-9046-82c4d4f19231"). InnerVolumeSpecName "kube-api-access-pxtk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:44:31.040244 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.040220 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9157de31-170a-411a-9046-82c4d4f19231-istio-podinfo" pod "9157de31-170a-411a-9046-82c4d4f19231" (UID: "9157de31-170a-411a-9046-82c4d4f19231"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:44:31.040540 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.040476 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9157de31-170a-411a-9046-82c4d4f19231-istio-token" pod "9157de31-170a-411a-9046-82c4d4f19231" (UID: "9157de31-170a-411a-9046-82c4d4f19231"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:44:31.040540 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.040511 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e4b4d14-b581-4844-a80e-a0edd473599e-kube-api-access-hrrpv" pod "5e4b4d14-b581-4844-a80e-a0edd473599e" (UID: "5e4b4d14-b581-4844-a80e-a0edd473599e"). InnerVolumeSpecName "kube-api-access-hrrpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:44:31.040685 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.040548 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-istio-envoy" pod "5e4b4d14-b581-4844-a80e-a0edd473599e" (UID: "5e4b4d14-b581-4844-a80e-a0edd473599e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:31.041930 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.041904 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-istio-envoy" pod "9157de31-170a-411a-9046-82c4d4f19231" (UID: "9157de31-170a-411a-9046-82c4d4f19231"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:31.123440 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.123398 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.123440 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.123443 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrrpv\" (UniqueName: \"kubernetes.io/projected/5e4b4d14-b581-4844-a80e-a0edd473599e-kube-api-access-hrrpv\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.123636 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.123461 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.123636 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.123475 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.123636 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.123487 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.123636 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.123500 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.123636 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.123512 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.123636 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.123528 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9157de31-170a-411a-9046-82c4d4f19231-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.123636 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.123542 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9157de31-170a-411a-9046-82c4d4f19231-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.123636 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.123554 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.123636 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.123567 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.123636 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.123579 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e4b4d14-b581-4844-a80e-a0edd473599e-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.123636 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.123592 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9157de31-170a-411a-9046-82c4d4f19231-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.123636 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.123605 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e4b4d14-b581-4844-a80e-a0edd473599e-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.123636 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.123621 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e4b4d14-b581-4844-a80e-a0edd473599e-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.123636 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.123633 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9157de31-170a-411a-9046-82c4d4f19231-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.124154 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.123644 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e4b4d14-b581-4844-a80e-a0edd473599e-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.124154 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.123657 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxtk5\" (UniqueName: \"kubernetes.io/projected/9157de31-170a-411a-9046-82c4d4f19231-kube-api-access-pxtk5\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.280083 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.280058 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:44:31.335533 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.335506 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istio-token\" (UniqueName: \"kubernetes.io/projected/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istio-token\") pod \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " Oct 30 17:44:31.335708 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.335600 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istio-data\") pod \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " Oct 30 17:44:31.335708 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.335682 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istiod-ca-cert\") pod \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " Oct 30 17:44:31.335832 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.335745 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e25ba26-3a38-490c-bc1a-199cfdc132a9-kube-api-access-4gb9n\" (UniqueName: \"kubernetes.io/projected/5e25ba26-3a38-490c-bc1a-199cfdc132a9-kube-api-access-4gb9n\") pod \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " Oct 30 17:44:31.335832 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.335794 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-workload-certs\") pod \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " Oct 30 17:44:31.335986 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.335958 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istio-data" pod "5e25ba26-3a38-490c-bc1a-199cfdc132a9" (UID: "5e25ba26-3a38-490c-bc1a-199cfdc132a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:31.336083 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.336061 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-workload-certs" pod "5e25ba26-3a38-490c-bc1a-199cfdc132a9" (UID: "5e25ba26-3a38-490c-bc1a-199cfdc132a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:31.336149 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.336122 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-credential-socket\") pod \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " Oct 30 17:44:31.336398 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.336373 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-credential-socket" pod "5e25ba26-3a38-490c-bc1a-199cfdc132a9" (UID: "5e25ba26-3a38-490c-bc1a-199cfdc132a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:31.336476 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.336395 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istiod-ca-cert" pod "5e25ba26-3a38-490c-bc1a-199cfdc132a9" (UID: "5e25ba26-3a38-490c-bc1a-199cfdc132a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:44:31.338073 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.338049 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-workload-socket\") pod \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " Oct 30 17:44:31.338235 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.338080 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e25ba26-3a38-490c-bc1a-199cfdc132a9-kube-api-access-4gb9n" pod "5e25ba26-3a38-490c-bc1a-199cfdc132a9" (UID: "5e25ba26-3a38-490c-bc1a-199cfdc132a9"). InnerVolumeSpecName "kube-api-access-4gb9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:44:31.338235 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.338087 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istio-podinfo\") pod \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " Oct 30 17:44:31.338235 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.338168 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istio-token" pod "5e25ba26-3a38-490c-bc1a-199cfdc132a9" (UID: "5e25ba26-3a38-490c-bc1a-199cfdc132a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:44:31.338235 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.338171 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istio-envoy\") pod \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\" (UID: \"5e25ba26-3a38-490c-bc1a-199cfdc132a9\") " Oct 30 17:44:31.338457 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.338273 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-workload-socket" pod "5e25ba26-3a38-490c-bc1a-199cfdc132a9" (UID: "5e25ba26-3a38-490c-bc1a-199cfdc132a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:31.338964 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.338939 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.339084 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.338966 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.339084 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.338981 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.339084 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.338995 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gb9n\" (UniqueName: \"kubernetes.io/projected/5e25ba26-3a38-490c-bc1a-199cfdc132a9-kube-api-access-4gb9n\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.339084 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.339007 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.339084 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.339020 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.339084 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.339034 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.340271 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.340245 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istio-podinfo" pod "5e25ba26-3a38-490c-bc1a-199cfdc132a9" (UID: "5e25ba26-3a38-490c-bc1a-199cfdc132a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:44:31.340356 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.340311 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istio-envoy" pod "5e25ba26-3a38-490c-bc1a-199cfdc132a9" (UID: "5e25ba26-3a38-490c-bc1a-199cfdc132a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:31.441400 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.441316 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.441400 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.441347 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e25ba26-3a38-490c-bc1a-199cfdc132a9-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.751845 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.751766 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" Oct 30 17:44:31.752330 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.751769 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp" event={"ID":"5e4b4d14-b581-4844-a80e-a0edd473599e","Type":"ContainerDied","Data":"53d588643f66139aedfb674d71910389e2b1ec123bc4374eb275abec1d99319b"} Oct 30 17:44:31.752330 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.751920 4965 scope.go:122] "RemoveContainer" containerID="0a4a72091f5c196bcdd521bfd19c5a7cdae4adbec5da81bc8389c2f65fda8f72" Oct 30 17:44:31.753356 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.753333 4965 generic.go:356] "Generic (PLEG): container finished" podID="5e25ba26-3a38-490c-bc1a-199cfdc132a9" containerID="2cc49f3d03304661d4f2f2ba290b890c835ef30e7ec4d107a5de327d435b1bc3" exitCode=0 Oct 30 17:44:31.753441 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.753380 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" Oct 30 17:44:31.753441 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.753411 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" event={"ID":"5e25ba26-3a38-490c-bc1a-199cfdc132a9","Type":"ContainerDied","Data":"2cc49f3d03304661d4f2f2ba290b890c835ef30e7ec4d107a5de327d435b1bc3"} Oct 30 17:44:31.753523 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.753459 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p" event={"ID":"5e25ba26-3a38-490c-bc1a-199cfdc132a9","Type":"ContainerDied","Data":"2c69aea38e4be741b7b71379d5bf4ad8d026e0302ab5e7cbf008a3d056216509"} Oct 30 17:44:31.755106 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.755085 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" Oct 30 17:44:31.755223 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.755088 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg" event={"ID":"9157de31-170a-411a-9046-82c4d4f19231","Type":"ContainerDied","Data":"2c0da5d21a341f64ebfad65594a49edd6c10d0d9bf12cbdbb8704527bc603888"} Oct 30 17:44:31.766714 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.766696 4965 scope.go:122] "RemoveContainer" containerID="2cc49f3d03304661d4f2f2ba290b890c835ef30e7ec4d107a5de327d435b1bc3" Oct 30 17:44:31.777099 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.777081 4965 scope.go:122] "RemoveContainer" containerID="2cc49f3d03304661d4f2f2ba290b890c835ef30e7ec4d107a5de327d435b1bc3" Oct 30 17:44:31.777380 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:44:31.777359 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cc49f3d03304661d4f2f2ba290b890c835ef30e7ec4d107a5de327d435b1bc3\": container with ID starting with 2cc49f3d03304661d4f2f2ba290b890c835ef30e7ec4d107a5de327d435b1bc3 not found: ID does not exist" containerID="2cc49f3d03304661d4f2f2ba290b890c835ef30e7ec4d107a5de327d435b1bc3" Oct 30 17:44:31.777458 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.777391 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cc49f3d03304661d4f2f2ba290b890c835ef30e7ec4d107a5de327d435b1bc3"} err="failed to get container status \"2cc49f3d03304661d4f2f2ba290b890c835ef30e7ec4d107a5de327d435b1bc3\": rpc error: code = NotFound desc = could not find container \"2cc49f3d03304661d4f2f2ba290b890c835ef30e7ec4d107a5de327d435b1bc3\": container with ID starting with 2cc49f3d03304661d4f2f2ba290b890c835ef30e7ec4d107a5de327d435b1bc3 not found: ID does not exist" Oct 30 17:44:31.777458 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.777418 4965 scope.go:122] "RemoveContainer" containerID="a60c2596d9f7ba2f106503fcc14b1ebfa0b55eabd53cd362f829ca0a02e613ab" Oct 30 17:44:31.826791 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.826761 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p"] Oct 30 17:44:31.838775 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.838749 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-rzf7p"] Oct 30 17:44:31.877387 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.877363 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg"] Oct 30 17:44:31.892423 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.892394 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-cgvsg"] Oct 30 17:44:31.931921 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.931851 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp"] Oct 30 17:44:31.935971 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:31.935950 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-tnbsp"] Oct 30 17:44:32.102244 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:32.102213 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e25ba26-3a38-490c-bc1a-199cfdc132a9" path="/var/lib/kubelet/pods/5e25ba26-3a38-490c-bc1a-199cfdc132a9/volumes" Oct 30 17:44:32.102753 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:32.102737 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e4b4d14-b581-4844-a80e-a0edd473599e" path="/var/lib/kubelet/pods/5e4b4d14-b581-4844-a80e-a0edd473599e/volumes" Oct 30 17:44:32.103252 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:32.103231 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9157de31-170a-411a-9046-82c4d4f19231" path="/var/lib/kubelet/pods/9157de31-170a-411a-9046-82c4d4f19231/volumes" Oct 30 17:44:32.300851 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:32.300815 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p"] Oct 30 17:44:32.302076 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:32.302055 4965 cpu_manager.go:414] "containerMap: removing container" podUID="9157de31-170a-411a-9046-82c4d4f19231" containerName="istio-proxy" Oct 30 17:44:32.302076 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:32.302077 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9157de31-170a-411a-9046-82c4d4f19231" containerName="istio-proxy" Oct 30 17:44:32.302250 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:32.302088 4965 cpu_manager.go:414] "containerMap: removing container" podUID="5e4b4d14-b581-4844-a80e-a0edd473599e" containerName="istio-proxy" Oct 30 17:44:32.302250 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:32.302093 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e4b4d14-b581-4844-a80e-a0edd473599e" containerName="istio-proxy" Oct 30 17:44:32.302250 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:32.302111 4965 cpu_manager.go:414] "containerMap: removing container" podUID="7be33b0f-8e47-4fcc-a091-8aaec48cd6bd" containerName="payload-processing" Oct 30 17:44:32.302250 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:32.302116 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7be33b0f-8e47-4fcc-a091-8aaec48cd6bd" containerName="payload-processing" Oct 30 17:44:32.302250 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:32.302150 4965 cpu_manager.go:414] "containerMap: removing container" podUID="5e25ba26-3a38-490c-bc1a-199cfdc132a9" containerName="istio-proxy" Oct 30 17:44:32.302250 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:32.302154 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e25ba26-3a38-490c-bc1a-199cfdc132a9" containerName="istio-proxy" Oct 30 17:44:32.302475 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:32.302280 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e25ba26-3a38-490c-bc1a-199cfdc132a9" containerName="istio-proxy" Oct 30 17:44:32.302475 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:32.302296 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="9157de31-170a-411a-9046-82c4d4f19231" containerName="istio-proxy" Oct 30 17:44:32.302475 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:32.302303 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e4b4d14-b581-4844-a80e-a0edd473599e" containerName="istio-proxy" Oct 30 17:44:32.302475 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:32.302312 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="7be33b0f-8e47-4fcc-a091-8aaec48cd6bd" containerName="payload-processing" Oct 30 17:44:32.307913 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:32.307895 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p" Oct 30 17:44:32.330280 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:32.330247 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-24fb0d-serving-cert\"" Oct 30 17:44:32.337517 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:32.337494 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p"] Oct 30 17:44:32.364545 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:32.364453 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9079b50c-c40f-4866-b177-414d6511ec8d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p\" (UID: \"9079b50c-c40f-4866-b177-414d6511ec8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p" Oct 30 17:44:32.364955 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:32.364927 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9079b50c-c40f-4866-b177-414d6511ec8d-metrics-certs\") pod \"maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p\" (UID: \"9079b50c-c40f-4866-b177-414d6511ec8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p" Oct 30 17:44:32.365149 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:32.365127 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfmjw\" (UniqueName: \"kubernetes.io/projected/9079b50c-c40f-4866-b177-414d6511ec8d-kube-api-access-pfmjw\") pod \"maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p\" (UID: \"9079b50c-c40f-4866-b177-414d6511ec8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p" Oct 30 17:44:32.365258 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:32.365243 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9079b50c-c40f-4866-b177-414d6511ec8d-maas-api-tls\") pod \"maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p\" (UID: \"9079b50c-c40f-4866-b177-414d6511ec8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p" Oct 30 17:44:32.466260 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:32.466225 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9079b50c-c40f-4866-b177-414d6511ec8d-metrics-certs\") pod \"maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p\" (UID: \"9079b50c-c40f-4866-b177-414d6511ec8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p" Oct 30 17:44:32.466436 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:32.466340 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfmjw\" (UniqueName: \"kubernetes.io/projected/9079b50c-c40f-4866-b177-414d6511ec8d-kube-api-access-pfmjw\") pod \"maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p\" (UID: \"9079b50c-c40f-4866-b177-414d6511ec8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p" Oct 30 17:44:32.466436 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:32.466399 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9079b50c-c40f-4866-b177-414d6511ec8d-maas-api-tls\") pod \"maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p\" (UID: \"9079b50c-c40f-4866-b177-414d6511ec8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p" Oct 30 17:44:32.466563 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:32.466473 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9079b50c-c40f-4866-b177-414d6511ec8d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p\" (UID: \"9079b50c-c40f-4866-b177-414d6511ec8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p" Oct 30 17:44:32.466721 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:44:32.466551 4965 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-24fb0d-serving-cert: secret "maas-api-e2e-inf-b-24fb0d-serving-cert" not found Oct 30 17:44:32.466878 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:44:32.466788 4965 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9079b50c-c40f-4866-b177-414d6511ec8d-maas-api-tls podName:9079b50c-c40f-4866-b177-414d6511ec8d nodeName:}" failed. No retries permitted until 2026-10-30 17:44:32.966764097 +0000 UTC m=+1931.018864028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9079b50c-c40f-4866-b177-414d6511ec8d-maas-api-tls") pod "maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p" (UID: "9079b50c-c40f-4866-b177-414d6511ec8d") : secret "maas-api-e2e-inf-b-24fb0d-serving-cert" not found Oct 30 17:44:32.467173 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:32.467147 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9079b50c-c40f-4866-b177-414d6511ec8d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p\" (UID: \"9079b50c-c40f-4866-b177-414d6511ec8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p" Oct 30 17:44:32.468927 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:32.468903 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9079b50c-c40f-4866-b177-414d6511ec8d-metrics-certs\") pod \"maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p\" (UID: \"9079b50c-c40f-4866-b177-414d6511ec8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p" Oct 30 17:44:32.494787 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:32.494760 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfmjw\" (UniqueName: \"kubernetes.io/projected/9079b50c-c40f-4866-b177-414d6511ec8d-kube-api-access-pfmjw\") pod \"maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p\" (UID: \"9079b50c-c40f-4866-b177-414d6511ec8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p" Oct 30 17:44:32.977513 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:32.977430 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9079b50c-c40f-4866-b177-414d6511ec8d-maas-api-tls\") pod \"maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p\" (UID: \"9079b50c-c40f-4866-b177-414d6511ec8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p" Oct 30 17:44:32.979925 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:32.979898 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9079b50c-c40f-4866-b177-414d6511ec8d-maas-api-tls\") pod \"maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p\" (UID: \"9079b50c-c40f-4866-b177-414d6511ec8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p" Oct 30 17:44:33.222608 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:33.222575 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p" Oct 30 17:44:33.536204 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:33.536172 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p"] Oct 30 17:44:33.536721 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:44:33.536695 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9079b50c_c40f_4866_b177_414d6511ec8d.slice/crio-aa21d5328d41cef33d010d0d86d86c6a14c470c7001a285b9e9ffa284d55a838 WatchSource:0}: Error finding container aa21d5328d41cef33d010d0d86d86c6a14c470c7001a285b9e9ffa284d55a838: Status 404 returned error can't find the container with id aa21d5328d41cef33d010d0d86d86c6a14c470c7001a285b9e9ffa284d55a838 Oct 30 17:44:33.619513 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:33.619481 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b"] Oct 30 17:44:33.625844 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:33.625821 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b" Oct 30 17:44:33.640501 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:33.640450 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bba90b\"/\"openshift-service-ca.crt\"" Oct 30 17:44:33.644366 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:33.644339 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b"] Oct 30 17:44:33.649424 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:33.649403 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bba90b\"/\"kube-root-ca.crt\"" Oct 30 17:44:33.649534 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:33.649453 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bba90b\"/\"default-dockercfg-bt8hh\"" Oct 30 17:44:33.652500 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:33.652477 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bba90b\"/\"test-model-bba90b-kserve-self-signed-certs\"" Oct 30 17:44:33.697211 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:33.697177 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f458064-ff24-4cf6-acb5-75830949042a-dshm\") pod \"test-model-bba90b-kserve-74557d96cf-cfc9b\" (UID: \"4f458064-ff24-4cf6-acb5-75830949042a\") " pod="ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b" Oct 30 17:44:33.697393 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:33.697307 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f458064-ff24-4cf6-acb5-75830949042a-tls-certs\") pod \"test-model-bba90b-kserve-74557d96cf-cfc9b\" (UID: \"4f458064-ff24-4cf6-acb5-75830949042a\") " pod="ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b" Oct 30 17:44:33.697393 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:33.697356 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcv4j\" (UniqueName: \"kubernetes.io/projected/4f458064-ff24-4cf6-acb5-75830949042a-kube-api-access-lcv4j\") pod \"test-model-bba90b-kserve-74557d96cf-cfc9b\" (UID: \"4f458064-ff24-4cf6-acb5-75830949042a\") " pod="ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b" Oct 30 17:44:33.697513 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:33.697399 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f458064-ff24-4cf6-acb5-75830949042a-home\") pod \"test-model-bba90b-kserve-74557d96cf-cfc9b\" (UID: \"4f458064-ff24-4cf6-acb5-75830949042a\") " pod="ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b" Oct 30 17:44:33.697568 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:33.697536 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f458064-ff24-4cf6-acb5-75830949042a-tmp-dir\") pod \"test-model-bba90b-kserve-74557d96cf-cfc9b\" (UID: \"4f458064-ff24-4cf6-acb5-75830949042a\") " pod="ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b" Oct 30 17:44:33.697568 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:33.697561 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f458064-ff24-4cf6-acb5-75830949042a-model-cache\") pod \"test-model-bba90b-kserve-74557d96cf-cfc9b\" (UID: \"4f458064-ff24-4cf6-acb5-75830949042a\") " pod="ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b" Oct 30 17:44:33.774649 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:33.774614 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p" event={"ID":"9079b50c-c40f-4866-b177-414d6511ec8d","Type":"ContainerStarted","Data":"aa21d5328d41cef33d010d0d86d86c6a14c470c7001a285b9e9ffa284d55a838"} Oct 30 17:44:33.798490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:33.798426 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f458064-ff24-4cf6-acb5-75830949042a-dshm\") pod \"test-model-bba90b-kserve-74557d96cf-cfc9b\" (UID: \"4f458064-ff24-4cf6-acb5-75830949042a\") " pod="ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b" Oct 30 17:44:33.798645 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:33.798520 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f458064-ff24-4cf6-acb5-75830949042a-tls-certs\") pod \"test-model-bba90b-kserve-74557d96cf-cfc9b\" (UID: \"4f458064-ff24-4cf6-acb5-75830949042a\") " pod="ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b" Oct 30 17:44:33.798645 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:33.798581 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcv4j\" (UniqueName: \"kubernetes.io/projected/4f458064-ff24-4cf6-acb5-75830949042a-kube-api-access-lcv4j\") pod \"test-model-bba90b-kserve-74557d96cf-cfc9b\" (UID: \"4f458064-ff24-4cf6-acb5-75830949042a\") " pod="ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b" Oct 30 17:44:33.798645 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:33.798623 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f458064-ff24-4cf6-acb5-75830949042a-home\") pod \"test-model-bba90b-kserve-74557d96cf-cfc9b\" (UID: \"4f458064-ff24-4cf6-acb5-75830949042a\") " pod="ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b" Oct 30 17:44:33.798820 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:33.798756 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f458064-ff24-4cf6-acb5-75830949042a-tmp-dir\") pod \"test-model-bba90b-kserve-74557d96cf-cfc9b\" (UID: \"4f458064-ff24-4cf6-acb5-75830949042a\") " pod="ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b" Oct 30 17:44:33.798820 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:33.798781 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f458064-ff24-4cf6-acb5-75830949042a-model-cache\") pod \"test-model-bba90b-kserve-74557d96cf-cfc9b\" (UID: \"4f458064-ff24-4cf6-acb5-75830949042a\") " pod="ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b" Oct 30 17:44:33.799115 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:33.799082 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f458064-ff24-4cf6-acb5-75830949042a-home\") pod \"test-model-bba90b-kserve-74557d96cf-cfc9b\" (UID: \"4f458064-ff24-4cf6-acb5-75830949042a\") " pod="ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b" Oct 30 17:44:33.799236 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:33.799115 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f458064-ff24-4cf6-acb5-75830949042a-tmp-dir\") pod \"test-model-bba90b-kserve-74557d96cf-cfc9b\" (UID: \"4f458064-ff24-4cf6-acb5-75830949042a\") " pod="ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b" Oct 30 17:44:33.799236 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:33.799194 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f458064-ff24-4cf6-acb5-75830949042a-model-cache\") pod \"test-model-bba90b-kserve-74557d96cf-cfc9b\" (UID: \"4f458064-ff24-4cf6-acb5-75830949042a\") " pod="ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b" Oct 30 17:44:33.800819 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:33.800793 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f458064-ff24-4cf6-acb5-75830949042a-dshm\") pod \"test-model-bba90b-kserve-74557d96cf-cfc9b\" (UID: \"4f458064-ff24-4cf6-acb5-75830949042a\") " pod="ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b" Oct 30 17:44:33.801021 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:33.801005 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f458064-ff24-4cf6-acb5-75830949042a-tls-certs\") pod \"test-model-bba90b-kserve-74557d96cf-cfc9b\" (UID: \"4f458064-ff24-4cf6-acb5-75830949042a\") " pod="ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b" Oct 30 17:44:33.823791 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:33.823766 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcv4j\" (UniqueName: \"kubernetes.io/projected/4f458064-ff24-4cf6-acb5-75830949042a-kube-api-access-lcv4j\") pod \"test-model-bba90b-kserve-74557d96cf-cfc9b\" (UID: \"4f458064-ff24-4cf6-acb5-75830949042a\") " pod="ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b" Oct 30 17:44:33.949888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:33.947921 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b" Oct 30 17:44:34.449972 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:34.449939 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b"] Oct 30 17:44:34.864885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:34.864088 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b" event={"ID":"4f458064-ff24-4cf6-acb5-75830949042a","Type":"ContainerStarted","Data":"a3753c57584061b932dd703207ea84fc3e01ac35fb88ad3dc96eaed4aae24293"} Oct 30 17:44:34.864885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:34.864158 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b" event={"ID":"4f458064-ff24-4cf6-acb5-75830949042a","Type":"ContainerStarted","Data":"f3c9881ff8ed6abef36ec1332c74a7f9750e04fc04a3e9c7a5f3344db5ffe0e5"} Oct 30 17:44:34.883831 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:34.883797 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p" event={"ID":"9079b50c-c40f-4866-b177-414d6511ec8d","Type":"ContainerStarted","Data":"a57e53f8bedeca16ed83fdfad5d3744c11d444aa50ae2c96b5a9d828b347558e"} Oct 30 17:44:34.884608 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:34.884574 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p" Oct 30 17:44:34.918337 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:34.918287 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b" podStartSLOduration=1.9182742689999999 podStartE2EDuration="1.918274269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:44:34.916404392 +0000 UTC m=+1932.968504351" watchObservedRunningTime="2026-10-30 17:44:34.918274269 +0000 UTC m=+1932.970374194" Oct 30 17:44:34.950797 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:34.950762 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b" Oct 30 17:44:34.990534 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:34.990473 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p" podStartSLOduration=2.6284760350000003 podStartE2EDuration="2.990459914s" totalImagesPullingTime="361.983879ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:44:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:44:34.989424323 +0000 UTC m=+1933.041524268" watchObservedRunningTime="2026-10-30 17:44:34.990459914 +0000 UTC m=+1933.042559870" Oct 30 17:44:36.239969 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:44:36.238309 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:44:38.277680 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:38.277647 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v"] Oct 30 17:44:38.283006 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:38.282981 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" Oct 30 17:44:38.299515 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:38.299487 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v"] Oct 30 17:44:38.307102 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:38.307079 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-143041\"/\"openshift-service-ca.crt\"" Oct 30 17:44:38.307228 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:38.307134 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-143041\"/\"kube-root-ca.crt\"" Oct 30 17:44:38.315374 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:38.315354 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-143041\"/\"default-dockercfg-slwvt\"" Oct 30 17:44:38.315494 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:38.315436 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-143041\"/\"test-model-143041-kserve-self-signed-certs\"" Oct 30 17:44:38.375792 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:38.375759 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a75d1f38-e7f2-47bd-b59a-538ee5b12412-tls-certs\") pod \"test-model-143041-kserve-568689b445-5kz6v\" (UID: \"a75d1f38-e7f2-47bd-b59a-538ee5b12412\") " pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" Oct 30 17:44:38.376012 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:38.375815 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a75d1f38-e7f2-47bd-b59a-538ee5b12412-tmp-dir\") pod \"test-model-143041-kserve-568689b445-5kz6v\" (UID: \"a75d1f38-e7f2-47bd-b59a-538ee5b12412\") " pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" Oct 30 17:44:38.376012 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:38.375938 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkfk7\" (UniqueName: \"kubernetes.io/projected/a75d1f38-e7f2-47bd-b59a-538ee5b12412-kube-api-access-rkfk7\") pod \"test-model-143041-kserve-568689b445-5kz6v\" (UID: \"a75d1f38-e7f2-47bd-b59a-538ee5b12412\") " pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" Oct 30 17:44:38.376012 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:38.375969 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a75d1f38-e7f2-47bd-b59a-538ee5b12412-dshm\") pod \"test-model-143041-kserve-568689b445-5kz6v\" (UID: \"a75d1f38-e7f2-47bd-b59a-538ee5b12412\") " pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" Oct 30 17:44:38.376234 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:38.376214 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a75d1f38-e7f2-47bd-b59a-538ee5b12412-model-cache\") pod \"test-model-143041-kserve-568689b445-5kz6v\" (UID: \"a75d1f38-e7f2-47bd-b59a-538ee5b12412\") " pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" Oct 30 17:44:38.376311 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:38.376298 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a75d1f38-e7f2-47bd-b59a-538ee5b12412-home\") pod \"test-model-143041-kserve-568689b445-5kz6v\" (UID: \"a75d1f38-e7f2-47bd-b59a-538ee5b12412\") " pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" Oct 30 17:44:38.478337 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:38.478303 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a75d1f38-e7f2-47bd-b59a-538ee5b12412-tls-certs\") pod \"test-model-143041-kserve-568689b445-5kz6v\" (UID: \"a75d1f38-e7f2-47bd-b59a-538ee5b12412\") " pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" Oct 30 17:44:38.478547 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:38.478356 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a75d1f38-e7f2-47bd-b59a-538ee5b12412-tmp-dir\") pod \"test-model-143041-kserve-568689b445-5kz6v\" (UID: \"a75d1f38-e7f2-47bd-b59a-538ee5b12412\") " pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" Oct 30 17:44:38.478547 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:38.478395 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkfk7\" (UniqueName: \"kubernetes.io/projected/a75d1f38-e7f2-47bd-b59a-538ee5b12412-kube-api-access-rkfk7\") pod \"test-model-143041-kserve-568689b445-5kz6v\" (UID: \"a75d1f38-e7f2-47bd-b59a-538ee5b12412\") " pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" Oct 30 17:44:38.478547 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:38.478426 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a75d1f38-e7f2-47bd-b59a-538ee5b12412-dshm\") pod \"test-model-143041-kserve-568689b445-5kz6v\" (UID: \"a75d1f38-e7f2-47bd-b59a-538ee5b12412\") " pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" Oct 30 17:44:38.478722 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:38.478629 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a75d1f38-e7f2-47bd-b59a-538ee5b12412-model-cache\") pod \"test-model-143041-kserve-568689b445-5kz6v\" (UID: \"a75d1f38-e7f2-47bd-b59a-538ee5b12412\") " pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" Oct 30 17:44:38.478774 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:38.478739 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a75d1f38-e7f2-47bd-b59a-538ee5b12412-home\") pod \"test-model-143041-kserve-568689b445-5kz6v\" (UID: \"a75d1f38-e7f2-47bd-b59a-538ee5b12412\") " pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" Oct 30 17:44:38.478876 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:38.478826 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a75d1f38-e7f2-47bd-b59a-538ee5b12412-tmp-dir\") pod \"test-model-143041-kserve-568689b445-5kz6v\" (UID: \"a75d1f38-e7f2-47bd-b59a-538ee5b12412\") " pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" Oct 30 17:44:38.479053 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:38.479036 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a75d1f38-e7f2-47bd-b59a-538ee5b12412-model-cache\") pod \"test-model-143041-kserve-568689b445-5kz6v\" (UID: \"a75d1f38-e7f2-47bd-b59a-538ee5b12412\") " pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" Oct 30 17:44:38.479125 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:38.479100 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a75d1f38-e7f2-47bd-b59a-538ee5b12412-home\") pod \"test-model-143041-kserve-568689b445-5kz6v\" (UID: \"a75d1f38-e7f2-47bd-b59a-538ee5b12412\") " pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" Oct 30 17:44:38.480937 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:38.480909 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a75d1f38-e7f2-47bd-b59a-538ee5b12412-dshm\") pod \"test-model-143041-kserve-568689b445-5kz6v\" (UID: \"a75d1f38-e7f2-47bd-b59a-538ee5b12412\") " pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" Oct 30 17:44:38.481107 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:38.481089 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a75d1f38-e7f2-47bd-b59a-538ee5b12412-tls-certs\") pod \"test-model-143041-kserve-568689b445-5kz6v\" (UID: \"a75d1f38-e7f2-47bd-b59a-538ee5b12412\") " pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" Oct 30 17:44:38.501917 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:38.501893 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkfk7\" (UniqueName: \"kubernetes.io/projected/a75d1f38-e7f2-47bd-b59a-538ee5b12412-kube-api-access-rkfk7\") pod \"test-model-143041-kserve-568689b445-5kz6v\" (UID: \"a75d1f38-e7f2-47bd-b59a-538ee5b12412\") " pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" Oct 30 17:44:38.612886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:38.612000 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" Oct 30 17:44:39.204896 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:39.200513 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v"] Oct 30 17:44:40.076752 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:40.076705 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" event={"ID":"a75d1f38-e7f2-47bd-b59a-538ee5b12412","Type":"ContainerStarted","Data":"5dae539cd7fc6e802f9f059c421c37b5ab821f9b78421c395a923c9df4d9ed9b"} Oct 30 17:44:40.077253 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:40.076763 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" event={"ID":"a75d1f38-e7f2-47bd-b59a-538ee5b12412","Type":"ContainerStarted","Data":"14f357fe26380973d4d602a208dc0f2bf06f206ef049cee6531fbd083498f234"} Oct 30 17:44:40.127648 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:40.127592 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" podStartSLOduration=2.127577939 podStartE2EDuration="2.127577939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:44:40.122457565 +0000 UTC m=+1938.174557499" watchObservedRunningTime="2026-10-30 17:44:40.127577939 +0000 UTC m=+1938.179677870" Oct 30 17:44:40.613299 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:40.613260 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" Oct 30 17:44:40.900664 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:40.900588 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p" Oct 30 17:44:43.950209 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:43.950180 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b" Oct 30 17:44:43.963490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:43.963464 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b" Oct 30 17:44:44.234317 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:44.234226 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b" Oct 30 17:44:48.613838 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:48.613806 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" Oct 30 17:44:48.627446 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:48.627419 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" Oct 30 17:44:49.268711 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:49.268654 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" Oct 30 17:44:57.717930 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:44:57.717888 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-74762d-dd9c689f5-2wvwk"] Oct 30 17:45:00.212430 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.212396 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf"] Oct 30 17:45:00.251148 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.251107 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw"] Oct 30 17:45:00.251458 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.251439 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf" Oct 30 17:45:00.263242 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.263216 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf"] Oct 30 17:45:00.263521 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.263504 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw" Oct 30 17:45:00.269131 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.269104 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw"] Oct 30 17:45:00.284972 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.284946 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq"] Oct 30 17:45:00.296454 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.296427 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b"] Oct 30 17:45:00.297303 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.297269 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq" Oct 30 17:45:00.305440 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.305405 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd"] Oct 30 17:45:00.305707 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.305678 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b" Oct 30 17:45:00.307766 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.307742 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\"" Oct 30 17:45:00.309597 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.309059 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e59f243c-037f-4b74-8c60-c7fbbbc10c65-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw\" (UID: \"e59f243c-037f-4b74-8c60-c7fbbbc10c65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw" Oct 30 17:45:00.309597 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.309167 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqpjv\" (UniqueName: \"kubernetes.io/projected/e59f243c-037f-4b74-8c60-c7fbbbc10c65-kube-api-access-fqpjv\") pod \"maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw\" (UID: \"e59f243c-037f-4b74-8c60-c7fbbbc10c65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw" Oct 30 17:45:00.309597 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.309231 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5ns9\" (UniqueName: \"kubernetes.io/projected/2e5f291f-ff17-41bc-b87f-2beaaa451c13-kube-api-access-x5ns9\") pod \"maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf\" (UID: \"2e5f291f-ff17-41bc-b87f-2beaaa451c13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf" Oct 30 17:45:00.309597 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.309296 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e5f291f-ff17-41bc-b87f-2beaaa451c13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf\" (UID: \"2e5f291f-ff17-41bc-b87f-2beaaa451c13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf" Oct 30 17:45:00.314968 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.314943 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq"] Oct 30 17:45:00.315082 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.315057 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd" Oct 30 17:45:00.321251 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.321229 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b"] Oct 30 17:45:00.326313 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.326292 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd"] Oct 30 17:45:00.410608 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.410562 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e59f243c-037f-4b74-8c60-c7fbbbc10c65-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw\" (UID: \"e59f243c-037f-4b74-8c60-c7fbbbc10c65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw" Oct 30 17:45:00.410779 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.410682 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqpjv\" (UniqueName: \"kubernetes.io/projected/e59f243c-037f-4b74-8c60-c7fbbbc10c65-kube-api-access-fqpjv\") pod \"maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw\" (UID: \"e59f243c-037f-4b74-8c60-c7fbbbc10c65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw" Oct 30 17:45:00.410779 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.410716 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c6bj\" (UniqueName: \"kubernetes.io/projected/09351610-5b9d-418c-b902-0e8dcf597ab5-kube-api-access-4c6bj\") pod \"maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq\" (UID: \"09351610-5b9d-418c-b902-0e8dcf597ab5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq" Oct 30 17:45:00.410779 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.410771 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5ns9\" (UniqueName: \"kubernetes.io/projected/2e5f291f-ff17-41bc-b87f-2beaaa451c13-kube-api-access-x5ns9\") pod \"maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf\" (UID: \"2e5f291f-ff17-41bc-b87f-2beaaa451c13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf" Oct 30 17:45:00.410915 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.410800 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgwbv\" (UniqueName: \"kubernetes.io/projected/6fcbf5b5-34ce-408a-874b-29f6ea910d40-kube-api-access-rgwbv\") pod \"maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b\" (UID: \"6fcbf5b5-34ce-408a-874b-29f6ea910d40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b" Oct 30 17:45:00.410966 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.410919 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e5f291f-ff17-41bc-b87f-2beaaa451c13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf\" (UID: \"2e5f291f-ff17-41bc-b87f-2beaaa451c13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf" Oct 30 17:45:00.410966 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.410949 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09351610-5b9d-418c-b902-0e8dcf597ab5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq\" (UID: \"09351610-5b9d-418c-b902-0e8dcf597ab5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq" Oct 30 17:45:00.411055 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.411024 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg85s\" (UniqueName: \"kubernetes.io/projected/058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5-kube-api-access-gg85s\") pod \"maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd\" (UID: \"058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd" Oct 30 17:45:00.411225 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.411202 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fcbf5b5-34ce-408a-874b-29f6ea910d40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b\" (UID: \"6fcbf5b5-34ce-408a-874b-29f6ea910d40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b" Oct 30 17:45:00.411282 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.411263 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd\" (UID: \"058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd" Oct 30 17:45:00.411366 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.411347 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e59f243c-037f-4b74-8c60-c7fbbbc10c65-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw\" (UID: \"e59f243c-037f-4b74-8c60-c7fbbbc10c65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw" Oct 30 17:45:00.411545 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.411526 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e5f291f-ff17-41bc-b87f-2beaaa451c13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf\" (UID: \"2e5f291f-ff17-41bc-b87f-2beaaa451c13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf" Oct 30 17:45:00.437374 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.437339 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqpjv\" (UniqueName: \"kubernetes.io/projected/e59f243c-037f-4b74-8c60-c7fbbbc10c65-kube-api-access-fqpjv\") pod \"maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw\" (UID: \"e59f243c-037f-4b74-8c60-c7fbbbc10c65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw" Oct 30 17:45:00.437521 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.437347 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5ns9\" (UniqueName: \"kubernetes.io/projected/2e5f291f-ff17-41bc-b87f-2beaaa451c13-kube-api-access-x5ns9\") pod \"maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf\" (UID: \"2e5f291f-ff17-41bc-b87f-2beaaa451c13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf" Oct 30 17:45:00.512680 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.512603 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c6bj\" (UniqueName: \"kubernetes.io/projected/09351610-5b9d-418c-b902-0e8dcf597ab5-kube-api-access-4c6bj\") pod \"maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq\" (UID: \"09351610-5b9d-418c-b902-0e8dcf597ab5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq" Oct 30 17:45:00.512680 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.512660 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgwbv\" (UniqueName: \"kubernetes.io/projected/6fcbf5b5-34ce-408a-874b-29f6ea910d40-kube-api-access-rgwbv\") pod \"maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b\" (UID: \"6fcbf5b5-34ce-408a-874b-29f6ea910d40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b" Oct 30 17:45:00.512952 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.512703 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09351610-5b9d-418c-b902-0e8dcf597ab5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq\" (UID: \"09351610-5b9d-418c-b902-0e8dcf597ab5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq" Oct 30 17:45:00.512952 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.512758 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg85s\" (UniqueName: \"kubernetes.io/projected/058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5-kube-api-access-gg85s\") pod \"maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd\" (UID: \"058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd" Oct 30 17:45:00.512952 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.512851 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fcbf5b5-34ce-408a-874b-29f6ea910d40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b\" (UID: \"6fcbf5b5-34ce-408a-874b-29f6ea910d40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b" Oct 30 17:45:00.512952 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.512914 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd\" (UID: \"058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd" Oct 30 17:45:00.513469 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.513449 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09351610-5b9d-418c-b902-0e8dcf597ab5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq\" (UID: \"09351610-5b9d-418c-b902-0e8dcf597ab5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq" Oct 30 17:45:00.513469 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.513460 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fcbf5b5-34ce-408a-874b-29f6ea910d40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b\" (UID: \"6fcbf5b5-34ce-408a-874b-29f6ea910d40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b" Oct 30 17:45:00.513551 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.513503 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd\" (UID: \"058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd" Oct 30 17:45:00.537560 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.537533 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg85s\" (UniqueName: \"kubernetes.io/projected/058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5-kube-api-access-gg85s\") pod \"maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd\" (UID: \"058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd" Oct 30 17:45:00.537694 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.537672 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c6bj\" (UniqueName: \"kubernetes.io/projected/09351610-5b9d-418c-b902-0e8dcf597ab5-kube-api-access-4c6bj\") pod \"maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq\" (UID: \"09351610-5b9d-418c-b902-0e8dcf597ab5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq" Oct 30 17:45:00.537757 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.537737 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgwbv\" (UniqueName: \"kubernetes.io/projected/6fcbf5b5-34ce-408a-874b-29f6ea910d40-kube-api-access-rgwbv\") pod \"maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b\" (UID: \"6fcbf5b5-34ce-408a-874b-29f6ea910d40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b" Oct 30 17:45:00.611528 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.611496 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf" Oct 30 17:45:00.628492 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.628470 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw" Oct 30 17:45:00.634680 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.634658 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq" Oct 30 17:45:00.651076 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.651044 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b" Oct 30 17:45:00.658269 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.658219 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd" Oct 30 17:45:00.937089 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.937056 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf"] Oct 30 17:45:00.941039 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:45:00.940999 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e5f291f_ff17_41bc_b87f_2beaaa451c13.slice/crio-2dfd728d3309c4b9783ff984cfbf7776c51575a6a0f2f6c1ffd739cf07b549d2 WatchSource:0}: Error finding container 2dfd728d3309c4b9783ff984cfbf7776c51575a6a0f2f6c1ffd739cf07b549d2: Status 404 returned error can't find the container with id 2dfd728d3309c4b9783ff984cfbf7776c51575a6a0f2f6c1ffd739cf07b549d2 Oct 30 17:45:00.981666 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.981644 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw"] Oct 30 17:45:00.981839 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:45:00.981715 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09351610_5b9d_418c_b902_0e8dcf597ab5.slice/crio-001da2a36d697a37e8ea23d2ccedbc4658db95d6d17e531148376fe225e4e26a WatchSource:0}: Error finding container 001da2a36d697a37e8ea23d2ccedbc4658db95d6d17e531148376fe225e4e26a: Status 404 returned error can't find the container with id 001da2a36d697a37e8ea23d2ccedbc4658db95d6d17e531148376fe225e4e26a Oct 30 17:45:00.988556 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.988534 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq"] Oct 30 17:45:00.998696 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:00.998673 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b"] Oct 30 17:45:01.016562 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:01.016533 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd"] Oct 30 17:45:01.016751 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:45:01.016718 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod058c2414_ea11_4cd1_89e1_f4b7cb8bc0b5.slice/crio-a96c3bd9b9c016d5835da8f26d21b72dd1224c03cce282c0f1c6be3c26832cd2 WatchSource:0}: Error finding container a96c3bd9b9c016d5835da8f26d21b72dd1224c03cce282c0f1c6be3c26832cd2: Status 404 returned error can't find the container with id a96c3bd9b9c016d5835da8f26d21b72dd1224c03cce282c0f1c6be3c26832cd2 Oct 30 17:45:01.505291 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:01.505244 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf" event={"ID":"2e5f291f-ff17-41bc-b87f-2beaaa451c13","Type":"ContainerStarted","Data":"2dfd728d3309c4b9783ff984cfbf7776c51575a6a0f2f6c1ffd739cf07b549d2"} Oct 30 17:45:01.506818 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:01.506790 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd" event={"ID":"058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5","Type":"ContainerStarted","Data":"a96c3bd9b9c016d5835da8f26d21b72dd1224c03cce282c0f1c6be3c26832cd2"} Oct 30 17:45:01.508372 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:01.508347 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b" event={"ID":"6fcbf5b5-34ce-408a-874b-29f6ea910d40","Type":"ContainerStarted","Data":"c696fe5c446ac32a3dcb45e9dc56f1745cb80c741a6ac22ec9e37fa6fdf0925e"} Oct 30 17:45:01.509661 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:01.509637 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq" event={"ID":"09351610-5b9d-418c-b902-0e8dcf597ab5","Type":"ContainerStarted","Data":"001da2a36d697a37e8ea23d2ccedbc4658db95d6d17e531148376fe225e4e26a"} Oct 30 17:45:01.510817 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:01.510798 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw" event={"ID":"e59f243c-037f-4b74-8c60-c7fbbbc10c65","Type":"ContainerStarted","Data":"a61a024f8ac2f0ed7b5cf6812f66f9f25877407fa48ab4078cee60c68532ec28"} Oct 30 17:45:02.720538 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:02.720488 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-74762d-dd9c689f5-2wvwk" podUID="931d5477-513c-41d3-93ec-e1f24b4ff8f4" containerName="payload-pre-processing" containerID="cri-o://18b4fb3a3db654e5a06afed2c1c68a984c3c71953e4625ac21c33a23cfe53140" gracePeriod=25 Oct 30 17:45:03.153347 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:03.153301 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-74762d-dd9c689f5-2wvwk" Oct 30 17:45:03.313085 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:03.311638 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/931d5477-513c-41d3-93ec-e1f24b4ff8f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/931d5477-513c-41d3-93ec-e1f24b4ff8f4-plugins-config-volume\") pod \"931d5477-513c-41d3-93ec-e1f24b4ff8f4\" (UID: \"931d5477-513c-41d3-93ec-e1f24b4ff8f4\") " Oct 30 17:45:03.313085 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:03.311762 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/931d5477-513c-41d3-93ec-e1f24b4ff8f4-kube-api-access-7fx89\" (UniqueName: \"kubernetes.io/projected/931d5477-513c-41d3-93ec-e1f24b4ff8f4-kube-api-access-7fx89\") pod \"931d5477-513c-41d3-93ec-e1f24b4ff8f4\" (UID: \"931d5477-513c-41d3-93ec-e1f24b4ff8f4\") " Oct 30 17:45:03.313575 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:03.313531 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/931d5477-513c-41d3-93ec-e1f24b4ff8f4-plugins-config-volume" pod "931d5477-513c-41d3-93ec-e1f24b4ff8f4" (UID: "931d5477-513c-41d3-93ec-e1f24b4ff8f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:45:03.316747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:03.316708 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/931d5477-513c-41d3-93ec-e1f24b4ff8f4-kube-api-access-7fx89" pod "931d5477-513c-41d3-93ec-e1f24b4ff8f4" (UID: "931d5477-513c-41d3-93ec-e1f24b4ff8f4"). InnerVolumeSpecName "kube-api-access-7fx89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:45:03.420941 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:03.417441 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/931d5477-513c-41d3-93ec-e1f24b4ff8f4-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:45:03.420941 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:03.417493 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fx89\" (UniqueName: \"kubernetes.io/projected/931d5477-513c-41d3-93ec-e1f24b4ff8f4-kube-api-access-7fx89\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:45:03.541147 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:03.541099 4965 generic.go:356] "Generic (PLEG): container finished" podID="931d5477-513c-41d3-93ec-e1f24b4ff8f4" containerID="18b4fb3a3db654e5a06afed2c1c68a984c3c71953e4625ac21c33a23cfe53140" exitCode=0 Oct 30 17:45:03.541518 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:03.541160 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-74762d-dd9c689f5-2wvwk" Oct 30 17:45:03.541518 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:03.541185 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-74762d-dd9c689f5-2wvwk" event={"ID":"931d5477-513c-41d3-93ec-e1f24b4ff8f4","Type":"ContainerDied","Data":"18b4fb3a3db654e5a06afed2c1c68a984c3c71953e4625ac21c33a23cfe53140"} Oct 30 17:45:03.541518 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:03.541236 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-74762d-dd9c689f5-2wvwk" event={"ID":"931d5477-513c-41d3-93ec-e1f24b4ff8f4","Type":"ContainerDied","Data":"5a460e38627c9fb73070ffc2a204bbd19e3c6d987fa888e540af75f85a340373"} Oct 30 17:45:03.541518 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:03.541264 4965 scope.go:122] "RemoveContainer" containerID="18b4fb3a3db654e5a06afed2c1c68a984c3c71953e4625ac21c33a23cfe53140" Oct 30 17:45:03.565118 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:03.565094 4965 scope.go:122] "RemoveContainer" containerID="18b4fb3a3db654e5a06afed2c1c68a984c3c71953e4625ac21c33a23cfe53140" Oct 30 17:45:03.565490 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:45:03.565461 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18b4fb3a3db654e5a06afed2c1c68a984c3c71953e4625ac21c33a23cfe53140\": container with ID starting with 18b4fb3a3db654e5a06afed2c1c68a984c3c71953e4625ac21c33a23cfe53140 not found: ID does not exist" containerID="18b4fb3a3db654e5a06afed2c1c68a984c3c71953e4625ac21c33a23cfe53140" Oct 30 17:45:03.565557 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:03.565502 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18b4fb3a3db654e5a06afed2c1c68a984c3c71953e4625ac21c33a23cfe53140"} err="failed to get container status \"18b4fb3a3db654e5a06afed2c1c68a984c3c71953e4625ac21c33a23cfe53140\": rpc error: code = NotFound desc = could not find container \"18b4fb3a3db654e5a06afed2c1c68a984c3c71953e4625ac21c33a23cfe53140\": container with ID starting with 18b4fb3a3db654e5a06afed2c1c68a984c3c71953e4625ac21c33a23cfe53140 not found: ID does not exist" Oct 30 17:45:03.615279 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:03.615243 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-74762d-dd9c689f5-2wvwk"] Oct 30 17:45:03.622327 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:03.622131 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-74762d-dd9c689f5-2wvwk"] Oct 30 17:45:04.106904 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:04.106853 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="931d5477-513c-41d3-93ec-e1f24b4ff8f4" path="/var/lib/kubelet/pods/931d5477-513c-41d3-93ec-e1f24b4ff8f4/volumes" Oct 30 17:45:05.568565 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:05.568524 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd" event={"ID":"058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5","Type":"ContainerStarted","Data":"fa08901bd03ea938915aeb3bcb693b23d62f109c52805b8ef1b13db749d8fb40"} Oct 30 17:45:05.571266 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:05.571236 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b" event={"ID":"6fcbf5b5-34ce-408a-874b-29f6ea910d40","Type":"ContainerStarted","Data":"b6af688f1f78ddb0c0baaaaee298fe8d3e03a1529f420629601d48dc30c894b4"} Oct 30 17:45:05.573269 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:05.573244 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf" event={"ID":"2e5f291f-ff17-41bc-b87f-2beaaa451c13","Type":"ContainerStarted","Data":"3aa381c34aaf8f145d5c783e4f8f59501cf0cac1d73e2ee76232ec7ee05b3220"} Oct 30 17:45:05.664095 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:05.662770 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd" podStartSLOduration=1.914366231 podStartE2EDuration="5.662747906s" totalImagesPullingTime="3.748381675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:05.661249188 +0000 UTC m=+1963.713349120" watchObservedRunningTime="2026-10-30 17:45:05.662747906 +0000 UTC m=+1963.714847843" Oct 30 17:45:05.722167 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:05.721450 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf" podStartSLOduration=1.650843519 podStartE2EDuration="5.721429204s" totalImagesPullingTime="4.070585685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:05.720097178 +0000 UTC m=+1963.772197111" watchObservedRunningTime="2026-10-30 17:45:05.721429204 +0000 UTC m=+1963.773529136" Oct 30 17:45:05.763474 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:05.763430 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b" podStartSLOduration=1.744729271 podStartE2EDuration="5.763416774s" totalImagesPullingTime="4.018687503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:05.761397185 +0000 UTC m=+1963.813497136" watchObservedRunningTime="2026-10-30 17:45:05.763416774 +0000 UTC m=+1963.815516704" Oct 30 17:45:05.998753 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:45:05.998711 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:45:06.583578 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:06.583548 4965 generic.go:356] "Generic (PLEG): container finished" podID="6fcbf5b5-34ce-408a-874b-29f6ea910d40" containerID="b6af688f1f78ddb0c0baaaaee298fe8d3e03a1529f420629601d48dc30c894b4" exitCode=60 Oct 30 17:45:06.584066 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:06.583611 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b" event={"ID":"6fcbf5b5-34ce-408a-874b-29f6ea910d40","Type":"ContainerDied","Data":"b6af688f1f78ddb0c0baaaaee298fe8d3e03a1529f420629601d48dc30c894b4"} Oct 30 17:45:06.584124 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:06.584099 4965 scope.go:122] "RemoveContainer" containerID="b6af688f1f78ddb0c0baaaaee298fe8d3e03a1529f420629601d48dc30c894b4" Oct 30 17:45:06.585512 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:06.585493 4965 generic.go:356] "Generic (PLEG): container finished" podID="e59f243c-037f-4b74-8c60-c7fbbbc10c65" containerID="200b4379442c9be1ef2f13b2a98b3c33d9723f2d25e0b2d12b470a07cfbb1ab7" exitCode=60 Oct 30 17:45:06.585628 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:06.585578 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw" event={"ID":"e59f243c-037f-4b74-8c60-c7fbbbc10c65","Type":"ContainerDied","Data":"200b4379442c9be1ef2f13b2a98b3c33d9723f2d25e0b2d12b470a07cfbb1ab7"} Oct 30 17:45:06.585913 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:06.585893 4965 scope.go:122] "RemoveContainer" containerID="200b4379442c9be1ef2f13b2a98b3c33d9723f2d25e0b2d12b470a07cfbb1ab7" Oct 30 17:45:06.587382 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:06.587358 4965 generic.go:356] "Generic (PLEG): container finished" podID="2e5f291f-ff17-41bc-b87f-2beaaa451c13" containerID="3aa381c34aaf8f145d5c783e4f8f59501cf0cac1d73e2ee76232ec7ee05b3220" exitCode=60 Oct 30 17:45:06.587486 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:06.587442 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf" event={"ID":"2e5f291f-ff17-41bc-b87f-2beaaa451c13","Type":"ContainerDied","Data":"3aa381c34aaf8f145d5c783e4f8f59501cf0cac1d73e2ee76232ec7ee05b3220"} Oct 30 17:45:06.587687 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:06.587669 4965 scope.go:122] "RemoveContainer" containerID="3aa381c34aaf8f145d5c783e4f8f59501cf0cac1d73e2ee76232ec7ee05b3220" Oct 30 17:45:06.589910 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:06.589450 4965 generic.go:356] "Generic (PLEG): container finished" podID="058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5" containerID="fa08901bd03ea938915aeb3bcb693b23d62f109c52805b8ef1b13db749d8fb40" exitCode=60 Oct 30 17:45:06.589910 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:06.589478 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd" event={"ID":"058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5","Type":"ContainerDied","Data":"fa08901bd03ea938915aeb3bcb693b23d62f109c52805b8ef1b13db749d8fb40"} Oct 30 17:45:06.589910 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:06.589797 4965 scope.go:122] "RemoveContainer" containerID="fa08901bd03ea938915aeb3bcb693b23d62f109c52805b8ef1b13db749d8fb40" Oct 30 17:45:07.609799 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:07.609757 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b" event={"ID":"6fcbf5b5-34ce-408a-874b-29f6ea910d40","Type":"ContainerStarted","Data":"868f6d26112f8d569d83ca73721ee4973a16c375260e7061f9ff4b38c868e4e7"} Oct 30 17:45:07.610239 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:07.610150 4965 scope.go:122] "RemoveContainer" containerID="868f6d26112f8d569d83ca73721ee4973a16c375260e7061f9ff4b38c868e4e7" Oct 30 17:45:07.610615 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:45:07.610590 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b_odh-ai-gateway-infra(6fcbf5b5-34ce-408a-874b-29f6ea910d40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b" podUID="6fcbf5b5-34ce-408a-874b-29f6ea910d40" Oct 30 17:45:07.612079 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:07.612052 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw" event={"ID":"e59f243c-037f-4b74-8c60-c7fbbbc10c65","Type":"ContainerStarted","Data":"7b2e208c5fba7c14588e9b99205b0fabb5100569a737d9b82abced184bdacd35"} Oct 30 17:45:07.614323 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:07.614299 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf" event={"ID":"2e5f291f-ff17-41bc-b87f-2beaaa451c13","Type":"ContainerStarted","Data":"cfe471955662df1eb852900b683019d96af181e1fdc9766022f008d5497588d6"} Oct 30 17:45:07.616430 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:07.616407 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd" event={"ID":"058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5","Type":"ContainerStarted","Data":"32ca4dce62e988a0d1e3d30c124647e02877f9d8df8db564f897c50f58b63fec"} Oct 30 17:45:07.740915 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:07.740850 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw" podStartSLOduration=3.622823869 podStartE2EDuration="7.740835514s" totalImagesPullingTime="4.118011645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:07.738189273 +0000 UTC m=+1965.790289201" watchObservedRunningTime="2026-10-30 17:45:07.740835514 +0000 UTC m=+1965.792935445" Oct 30 17:45:08.627806 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:08.627766 4965 generic.go:356] "Generic (PLEG): container finished" podID="2e5f291f-ff17-41bc-b87f-2beaaa451c13" containerID="cfe471955662df1eb852900b683019d96af181e1fdc9766022f008d5497588d6" exitCode=60 Oct 30 17:45:08.628291 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:08.627846 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf" event={"ID":"2e5f291f-ff17-41bc-b87f-2beaaa451c13","Type":"ContainerDied","Data":"cfe471955662df1eb852900b683019d96af181e1fdc9766022f008d5497588d6"} Oct 30 17:45:08.628291 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:08.627935 4965 scope.go:122] "RemoveContainer" containerID="3aa381c34aaf8f145d5c783e4f8f59501cf0cac1d73e2ee76232ec7ee05b3220" Oct 30 17:45:08.628291 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:08.628209 4965 scope.go:122] "RemoveContainer" containerID="cfe471955662df1eb852900b683019d96af181e1fdc9766022f008d5497588d6" Oct 30 17:45:08.628718 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:45:08.628686 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf_odh-ai-gateway-infra(2e5f291f-ff17-41bc-b87f-2beaaa451c13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf" podUID="2e5f291f-ff17-41bc-b87f-2beaaa451c13" Oct 30 17:45:08.629779 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:08.629758 4965 generic.go:356] "Generic (PLEG): container finished" podID="058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5" containerID="32ca4dce62e988a0d1e3d30c124647e02877f9d8df8db564f897c50f58b63fec" exitCode=60 Oct 30 17:45:08.629906 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:08.629842 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd" event={"ID":"058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5","Type":"ContainerDied","Data":"32ca4dce62e988a0d1e3d30c124647e02877f9d8df8db564f897c50f58b63fec"} Oct 30 17:45:08.630115 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:08.630098 4965 scope.go:122] "RemoveContainer" containerID="32ca4dce62e988a0d1e3d30c124647e02877f9d8df8db564f897c50f58b63fec" Oct 30 17:45:08.630492 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:45:08.630465 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd_odh-ai-gateway-infra(058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd" podUID="058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5" Oct 30 17:45:08.631816 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:08.631800 4965 generic.go:356] "Generic (PLEG): container finished" podID="6fcbf5b5-34ce-408a-874b-29f6ea910d40" containerID="868f6d26112f8d569d83ca73721ee4973a16c375260e7061f9ff4b38c868e4e7" exitCode=60 Oct 30 17:45:08.631912 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:08.631850 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b" event={"ID":"6fcbf5b5-34ce-408a-874b-29f6ea910d40","Type":"ContainerDied","Data":"868f6d26112f8d569d83ca73721ee4973a16c375260e7061f9ff4b38c868e4e7"} Oct 30 17:45:08.632208 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:08.632181 4965 scope.go:122] "RemoveContainer" containerID="868f6d26112f8d569d83ca73721ee4973a16c375260e7061f9ff4b38c868e4e7" Oct 30 17:45:08.632535 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:45:08.632514 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b_odh-ai-gateway-infra(6fcbf5b5-34ce-408a-874b-29f6ea910d40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b" podUID="6fcbf5b5-34ce-408a-874b-29f6ea910d40" Oct 30 17:45:08.633570 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:08.633548 4965 generic.go:356] "Generic (PLEG): container finished" podID="09351610-5b9d-418c-b902-0e8dcf597ab5" containerID="1e25e03e5e8766b758383a6586cc718b75f5355122c50cc0bf655e7f026bead7" exitCode=60 Oct 30 17:45:08.633657 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:08.633613 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq" event={"ID":"09351610-5b9d-418c-b902-0e8dcf597ab5","Type":"ContainerDied","Data":"1e25e03e5e8766b758383a6586cc718b75f5355122c50cc0bf655e7f026bead7"} Oct 30 17:45:08.633880 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:08.633851 4965 scope.go:122] "RemoveContainer" containerID="1e25e03e5e8766b758383a6586cc718b75f5355122c50cc0bf655e7f026bead7" Oct 30 17:45:08.635601 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:08.635582 4965 generic.go:356] "Generic (PLEG): container finished" podID="e59f243c-037f-4b74-8c60-c7fbbbc10c65" containerID="7b2e208c5fba7c14588e9b99205b0fabb5100569a737d9b82abced184bdacd35" exitCode=60 Oct 30 17:45:08.635704 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:08.635668 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw" event={"ID":"e59f243c-037f-4b74-8c60-c7fbbbc10c65","Type":"ContainerDied","Data":"7b2e208c5fba7c14588e9b99205b0fabb5100569a737d9b82abced184bdacd35"} Oct 30 17:45:08.635912 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:08.635895 4965 scope.go:122] "RemoveContainer" containerID="7b2e208c5fba7c14588e9b99205b0fabb5100569a737d9b82abced184bdacd35" Oct 30 17:45:08.636265 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:45:08.636143 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw_odh-ai-gateway-infra(e59f243c-037f-4b74-8c60-c7fbbbc10c65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw" podUID="e59f243c-037f-4b74-8c60-c7fbbbc10c65" Oct 30 17:45:08.642183 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:08.642166 4965 scope.go:122] "RemoveContainer" containerID="fa08901bd03ea938915aeb3bcb693b23d62f109c52805b8ef1b13db749d8fb40" Oct 30 17:45:08.656694 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:08.656675 4965 scope.go:122] "RemoveContainer" containerID="b6af688f1f78ddb0c0baaaaee298fe8d3e03a1529f420629601d48dc30c894b4" Oct 30 17:45:08.681088 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:08.681066 4965 scope.go:122] "RemoveContainer" containerID="200b4379442c9be1ef2f13b2a98b3c33d9723f2d25e0b2d12b470a07cfbb1ab7" Oct 30 17:45:09.319931 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.319778 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2"] Oct 30 17:45:09.321260 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.321229 4965 cpu_manager.go:414] "containerMap: removing container" podUID="931d5477-513c-41d3-93ec-e1f24b4ff8f4" containerName="payload-pre-processing" Oct 30 17:45:09.321260 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.321257 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="931d5477-513c-41d3-93ec-e1f24b4ff8f4" containerName="payload-pre-processing" Oct 30 17:45:09.321567 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.321549 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="931d5477-513c-41d3-93ec-e1f24b4ff8f4" containerName="payload-pre-processing" Oct 30 17:45:09.326724 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.326698 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" Oct 30 17:45:09.343382 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.343197 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-38f445\"/\"openshift-service-ca.crt\"" Oct 30 17:45:09.347937 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.347915 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-38f445\"/\"kube-root-ca.crt\"" Oct 30 17:45:09.348272 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.348257 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-38f445\"/\"test-model-38f445-kserve-self-signed-certs\"" Oct 30 17:45:09.348782 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.348419 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-38f445\"/\"default-dockercfg-v4sr4\"" Oct 30 17:45:09.348782 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.348279 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2"] Oct 30 17:45:09.459482 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.459451 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfvnr\" (UniqueName: \"kubernetes.io/projected/4ea02e31-5a9d-4070-8227-4e71fb8e849e-kube-api-access-bfvnr\") pod \"test-model-38f445-kserve-6d7ff8c99f-nt8g2\" (UID: \"4ea02e31-5a9d-4070-8227-4e71fb8e849e\") " pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" Oct 30 17:45:09.459655 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.459561 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ea02e31-5a9d-4070-8227-4e71fb8e849e-model-cache\") pod \"test-model-38f445-kserve-6d7ff8c99f-nt8g2\" (UID: \"4ea02e31-5a9d-4070-8227-4e71fb8e849e\") " pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" Oct 30 17:45:09.459655 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.459599 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ea02e31-5a9d-4070-8227-4e71fb8e849e-home\") pod \"test-model-38f445-kserve-6d7ff8c99f-nt8g2\" (UID: \"4ea02e31-5a9d-4070-8227-4e71fb8e849e\") " pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" Oct 30 17:45:09.459735 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.459683 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ea02e31-5a9d-4070-8227-4e71fb8e849e-dshm\") pod \"test-model-38f445-kserve-6d7ff8c99f-nt8g2\" (UID: \"4ea02e31-5a9d-4070-8227-4e71fb8e849e\") " pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" Oct 30 17:45:09.459853 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.459838 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea02e31-5a9d-4070-8227-4e71fb8e849e-tmp-dir\") pod \"test-model-38f445-kserve-6d7ff8c99f-nt8g2\" (UID: \"4ea02e31-5a9d-4070-8227-4e71fb8e849e\") " pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" Oct 30 17:45:09.459917 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.459904 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ea02e31-5a9d-4070-8227-4e71fb8e849e-tls-certs\") pod \"test-model-38f445-kserve-6d7ff8c99f-nt8g2\" (UID: \"4ea02e31-5a9d-4070-8227-4e71fb8e849e\") " pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" Oct 30 17:45:09.588708 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.588651 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ea02e31-5a9d-4070-8227-4e71fb8e849e-tls-certs\") pod \"test-model-38f445-kserve-6d7ff8c99f-nt8g2\" (UID: \"4ea02e31-5a9d-4070-8227-4e71fb8e849e\") " pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" Oct 30 17:45:09.620943 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.590035 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfvnr\" (UniqueName: \"kubernetes.io/projected/4ea02e31-5a9d-4070-8227-4e71fb8e849e-kube-api-access-bfvnr\") pod \"test-model-38f445-kserve-6d7ff8c99f-nt8g2\" (UID: \"4ea02e31-5a9d-4070-8227-4e71fb8e849e\") " pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" Oct 30 17:45:09.641987 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.629026 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ea02e31-5a9d-4070-8227-4e71fb8e849e-model-cache\") pod \"test-model-38f445-kserve-6d7ff8c99f-nt8g2\" (UID: \"4ea02e31-5a9d-4070-8227-4e71fb8e849e\") " pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" Oct 30 17:45:09.641987 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.629091 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ea02e31-5a9d-4070-8227-4e71fb8e849e-home\") pod \"test-model-38f445-kserve-6d7ff8c99f-nt8g2\" (UID: \"4ea02e31-5a9d-4070-8227-4e71fb8e849e\") " pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" Oct 30 17:45:09.641987 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.629175 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ea02e31-5a9d-4070-8227-4e71fb8e849e-dshm\") pod \"test-model-38f445-kserve-6d7ff8c99f-nt8g2\" (UID: \"4ea02e31-5a9d-4070-8227-4e71fb8e849e\") " pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" Oct 30 17:45:09.641987 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.629421 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea02e31-5a9d-4070-8227-4e71fb8e849e-tmp-dir\") pod \"test-model-38f445-kserve-6d7ff8c99f-nt8g2\" (UID: \"4ea02e31-5a9d-4070-8227-4e71fb8e849e\") " pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" Oct 30 17:45:09.641987 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.629918 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea02e31-5a9d-4070-8227-4e71fb8e849e-tmp-dir\") pod \"test-model-38f445-kserve-6d7ff8c99f-nt8g2\" (UID: \"4ea02e31-5a9d-4070-8227-4e71fb8e849e\") " pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" Oct 30 17:45:09.641987 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.594838 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ea02e31-5a9d-4070-8227-4e71fb8e849e-tls-certs\") pod \"test-model-38f445-kserve-6d7ff8c99f-nt8g2\" (UID: \"4ea02e31-5a9d-4070-8227-4e71fb8e849e\") " pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" Oct 30 17:45:09.641987 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.630200 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ea02e31-5a9d-4070-8227-4e71fb8e849e-model-cache\") pod \"test-model-38f445-kserve-6d7ff8c99f-nt8g2\" (UID: \"4ea02e31-5a9d-4070-8227-4e71fb8e849e\") " pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" Oct 30 17:45:09.641987 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.630437 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ea02e31-5a9d-4070-8227-4e71fb8e849e-home\") pod \"test-model-38f445-kserve-6d7ff8c99f-nt8g2\" (UID: \"4ea02e31-5a9d-4070-8227-4e71fb8e849e\") " pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" Oct 30 17:45:09.684552 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.680230 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ea02e31-5a9d-4070-8227-4e71fb8e849e-dshm\") pod \"test-model-38f445-kserve-6d7ff8c99f-nt8g2\" (UID: \"4ea02e31-5a9d-4070-8227-4e71fb8e849e\") " pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" Oct 30 17:45:09.710966 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.693598 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfvnr\" (UniqueName: \"kubernetes.io/projected/4ea02e31-5a9d-4070-8227-4e71fb8e849e-kube-api-access-bfvnr\") pod \"test-model-38f445-kserve-6d7ff8c99f-nt8g2\" (UID: \"4ea02e31-5a9d-4070-8227-4e71fb8e849e\") " pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" Oct 30 17:45:09.790152 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.789890 4965 scope.go:122] "RemoveContainer" containerID="868f6d26112f8d569d83ca73721ee4973a16c375260e7061f9ff4b38c868e4e7" Oct 30 17:45:09.790842 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:45:09.790811 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b_odh-ai-gateway-infra(6fcbf5b5-34ce-408a-874b-29f6ea910d40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b" podUID="6fcbf5b5-34ce-408a-874b-29f6ea910d40" Oct 30 17:45:09.795936 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.793887 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq" event={"ID":"09351610-5b9d-418c-b902-0e8dcf597ab5","Type":"ContainerStarted","Data":"50a74e40b65174210c0b3d0633914c8d35a40a14e6abd01010f9e68ef665b869"} Oct 30 17:45:09.798198 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.798162 4965 scope.go:122] "RemoveContainer" containerID="7b2e208c5fba7c14588e9b99205b0fabb5100569a737d9b82abced184bdacd35" Oct 30 17:45:09.798618 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:45:09.798591 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw_odh-ai-gateway-infra(e59f243c-037f-4b74-8c60-c7fbbbc10c65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw" podUID="e59f243c-037f-4b74-8c60-c7fbbbc10c65" Oct 30 17:45:09.800215 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.800188 4965 scope.go:122] "RemoveContainer" containerID="cfe471955662df1eb852900b683019d96af181e1fdc9766022f008d5497588d6" Oct 30 17:45:09.800506 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:45:09.800488 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf_odh-ai-gateway-infra(2e5f291f-ff17-41bc-b87f-2beaaa451c13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf" podUID="2e5f291f-ff17-41bc-b87f-2beaaa451c13" Oct 30 17:45:09.802020 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.802003 4965 scope.go:122] "RemoveContainer" containerID="32ca4dce62e988a0d1e3d30c124647e02877f9d8df8db564f897c50f58b63fec" Oct 30 17:45:09.802297 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:45:09.802280 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd_odh-ai-gateway-infra(058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd" podUID="058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5" Oct 30 17:45:09.838016 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.837967 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq" podStartSLOduration=3.452073048 podStartE2EDuration="9.837953634s" totalImagesPullingTime="6.385880586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:09.833788345 +0000 UTC m=+1967.885888278" watchObservedRunningTime="2026-10-30 17:45:09.837953634 +0000 UTC m=+1967.890053564" Oct 30 17:45:09.957716 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:09.957643 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" Oct 30 17:45:10.267011 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:45:10.266964 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ea02e31_5a9d_4070_8227_4e71fb8e849e.slice/crio-48be7d1feddca4a9b74eb2c8d27ba22f7f7e8a4f5d0f1513d06782788868625c WatchSource:0}: Error finding container 48be7d1feddca4a9b74eb2c8d27ba22f7f7e8a4f5d0f1513d06782788868625c: Status 404 returned error can't find the container with id 48be7d1feddca4a9b74eb2c8d27ba22f7f7e8a4f5d0f1513d06782788868625c Oct 30 17:45:10.267673 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:10.267627 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2"] Oct 30 17:45:10.816560 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:10.816525 4965 generic.go:356] "Generic (PLEG): container finished" podID="09351610-5b9d-418c-b902-0e8dcf597ab5" containerID="50a74e40b65174210c0b3d0633914c8d35a40a14e6abd01010f9e68ef665b869" exitCode=60 Oct 30 17:45:10.817070 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:10.816616 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq" event={"ID":"09351610-5b9d-418c-b902-0e8dcf597ab5","Type":"ContainerDied","Data":"50a74e40b65174210c0b3d0633914c8d35a40a14e6abd01010f9e68ef665b869"} Oct 30 17:45:10.817070 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:10.816676 4965 scope.go:122] "RemoveContainer" containerID="1e25e03e5e8766b758383a6586cc718b75f5355122c50cc0bf655e7f026bead7" Oct 30 17:45:10.817070 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:10.816969 4965 scope.go:122] "RemoveContainer" containerID="50a74e40b65174210c0b3d0633914c8d35a40a14e6abd01010f9e68ef665b869" Oct 30 17:45:10.817814 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:45:10.817420 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq_odh-ai-gateway-infra(09351610-5b9d-418c-b902-0e8dcf597ab5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq" podUID="09351610-5b9d-418c-b902-0e8dcf597ab5" Oct 30 17:45:10.819116 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:10.819094 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" event={"ID":"4ea02e31-5a9d-4070-8227-4e71fb8e849e","Type":"ContainerStarted","Data":"5054452fb18603905b533e7a3bfc328f4188eddd9560102cf503282b02b97a66"} Oct 30 17:45:10.819210 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:10.819123 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" event={"ID":"4ea02e31-5a9d-4070-8227-4e71fb8e849e","Type":"ContainerStarted","Data":"48be7d1feddca4a9b74eb2c8d27ba22f7f7e8a4f5d0f1513d06782788868625c"} Oct 30 17:45:10.913838 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:10.909040 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" podStartSLOduration=1.909020796 podStartE2EDuration="1.909020796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:10.897612696 +0000 UTC m=+1968.949712630" watchObservedRunningTime="2026-10-30 17:45:10.909020796 +0000 UTC m=+1968.961120732" Oct 30 17:45:10.958350 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:10.958320 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" Oct 30 17:45:11.829100 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:11.829068 4965 scope.go:122] "RemoveContainer" containerID="50a74e40b65174210c0b3d0633914c8d35a40a14e6abd01010f9e68ef665b869" Oct 30 17:45:11.829543 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:45:11.829519 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq_odh-ai-gateway-infra(09351610-5b9d-418c-b902-0e8dcf597ab5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq" podUID="09351610-5b9d-418c-b902-0e8dcf597ab5" Oct 30 17:45:18.037885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:18.028708 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:45:19.958396 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:19.958349 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" Oct 30 17:45:19.974765 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:19.974731 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" Oct 30 17:45:21.104885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:21.094813 4965 scope.go:122] "RemoveContainer" containerID="32ca4dce62e988a0d1e3d30c124647e02877f9d8df8db564f897c50f58b63fec" Oct 30 17:45:21.194193 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:21.193742 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" Oct 30 17:45:22.001661 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:22.001623 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd" event={"ID":"058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5","Type":"ContainerStarted","Data":"edfad6c40a9b5a8cf8b9542c07c9bf3a43f6f8ee8d9edd011b40a8ab46ee84e8"} Oct 30 17:45:22.096827 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:22.096788 4965 scope.go:122] "RemoveContainer" containerID="7b2e208c5fba7c14588e9b99205b0fabb5100569a737d9b82abced184bdacd35" Oct 30 17:45:22.097024 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:22.097000 4965 scope.go:122] "RemoveContainer" containerID="cfe471955662df1eb852900b683019d96af181e1fdc9766022f008d5497588d6" Oct 30 17:45:22.097202 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:22.097180 4965 scope.go:122] "RemoveContainer" containerID="868f6d26112f8d569d83ca73721ee4973a16c375260e7061f9ff4b38c868e4e7" Oct 30 17:45:23.014438 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:23.014399 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b" event={"ID":"6fcbf5b5-34ce-408a-874b-29f6ea910d40","Type":"ContainerStarted","Data":"ec2ae82be0fc542e6c2b0cf887fe98e6f1ff8103b2bcf13e06e23dc05264fc6e"} Oct 30 17:45:23.021585 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:23.017597 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw" event={"ID":"e59f243c-037f-4b74-8c60-c7fbbbc10c65","Type":"ContainerStarted","Data":"0f49d4fd10c6b1c82dc457606bf3b8006cb8ca5f8a971e6990a5a073bf199d7c"} Oct 30 17:45:23.021585 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:23.020557 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf" event={"ID":"2e5f291f-ff17-41bc-b87f-2beaaa451c13","Type":"ContainerStarted","Data":"a786716774b9cf3114dc6475ac88732a61d462d7bd0cd1ee5e96bd8389b56f46"} Oct 30 17:45:23.024488 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:23.024456 4965 generic.go:356] "Generic (PLEG): container finished" podID="058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5" containerID="edfad6c40a9b5a8cf8b9542c07c9bf3a43f6f8ee8d9edd011b40a8ab46ee84e8" exitCode=60 Oct 30 17:45:23.024641 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:23.024624 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd" event={"ID":"058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5","Type":"ContainerDied","Data":"edfad6c40a9b5a8cf8b9542c07c9bf3a43f6f8ee8d9edd011b40a8ab46ee84e8"} Oct 30 17:45:23.024711 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:23.024673 4965 scope.go:122] "RemoveContainer" containerID="32ca4dce62e988a0d1e3d30c124647e02877f9d8df8db564f897c50f58b63fec" Oct 30 17:45:23.025195 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:23.025171 4965 scope.go:122] "RemoveContainer" containerID="edfad6c40a9b5a8cf8b9542c07c9bf3a43f6f8ee8d9edd011b40a8ab46ee84e8" Oct 30 17:45:23.025622 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:45:23.025525 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd_odh-ai-gateway-infra(058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd" podUID="058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5" Oct 30 17:45:23.059252 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:23.059219 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd"] Oct 30 17:45:24.034240 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:24.034205 4965 generic.go:356] "Generic (PLEG): container finished" podID="2e5f291f-ff17-41bc-b87f-2beaaa451c13" containerID="a786716774b9cf3114dc6475ac88732a61d462d7bd0cd1ee5e96bd8389b56f46" exitCode=60 Oct 30 17:45:24.034712 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:24.034277 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf" event={"ID":"2e5f291f-ff17-41bc-b87f-2beaaa451c13","Type":"ContainerDied","Data":"a786716774b9cf3114dc6475ac88732a61d462d7bd0cd1ee5e96bd8389b56f46"} Oct 30 17:45:24.034712 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:24.034364 4965 scope.go:122] "RemoveContainer" containerID="cfe471955662df1eb852900b683019d96af181e1fdc9766022f008d5497588d6" Oct 30 17:45:24.034822 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:24.034741 4965 scope.go:122] "RemoveContainer" containerID="a786716774b9cf3114dc6475ac88732a61d462d7bd0cd1ee5e96bd8389b56f46" Oct 30 17:45:24.035180 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:45:24.035151 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf_odh-ai-gateway-infra(2e5f291f-ff17-41bc-b87f-2beaaa451c13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf" podUID="2e5f291f-ff17-41bc-b87f-2beaaa451c13" Oct 30 17:45:24.038094 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:24.038071 4965 generic.go:356] "Generic (PLEG): container finished" podID="6fcbf5b5-34ce-408a-874b-29f6ea910d40" containerID="ec2ae82be0fc542e6c2b0cf887fe98e6f1ff8103b2bcf13e06e23dc05264fc6e" exitCode=60 Oct 30 17:45:24.038224 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:24.038141 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b" event={"ID":"6fcbf5b5-34ce-408a-874b-29f6ea910d40","Type":"ContainerDied","Data":"ec2ae82be0fc542e6c2b0cf887fe98e6f1ff8103b2bcf13e06e23dc05264fc6e"} Oct 30 17:45:24.038551 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:24.038529 4965 scope.go:122] "RemoveContainer" containerID="ec2ae82be0fc542e6c2b0cf887fe98e6f1ff8103b2bcf13e06e23dc05264fc6e" Oct 30 17:45:24.038928 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:45:24.038907 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b_odh-ai-gateway-infra(6fcbf5b5-34ce-408a-874b-29f6ea910d40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b" podUID="6fcbf5b5-34ce-408a-874b-29f6ea910d40" Oct 30 17:45:24.040063 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:24.040042 4965 generic.go:356] "Generic (PLEG): container finished" podID="e59f243c-037f-4b74-8c60-c7fbbbc10c65" containerID="0f49d4fd10c6b1c82dc457606bf3b8006cb8ca5f8a971e6990a5a073bf199d7c" exitCode=60 Oct 30 17:45:24.040151 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:24.040099 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw" event={"ID":"e59f243c-037f-4b74-8c60-c7fbbbc10c65","Type":"ContainerDied","Data":"0f49d4fd10c6b1c82dc457606bf3b8006cb8ca5f8a971e6990a5a073bf199d7c"} Oct 30 17:45:24.040370 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:24.040356 4965 scope.go:122] "RemoveContainer" containerID="0f49d4fd10c6b1c82dc457606bf3b8006cb8ca5f8a971e6990a5a073bf199d7c" Oct 30 17:45:24.040648 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:45:24.040630 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw_odh-ai-gateway-infra(e59f243c-037f-4b74-8c60-c7fbbbc10c65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw" podUID="e59f243c-037f-4b74-8c60-c7fbbbc10c65" Oct 30 17:45:24.050779 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:24.050755 4965 scope.go:122] "RemoveContainer" containerID="868f6d26112f8d569d83ca73721ee4973a16c375260e7061f9ff4b38c868e4e7" Oct 30 17:45:24.067367 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:24.067348 4965 scope.go:122] "RemoveContainer" containerID="7b2e208c5fba7c14588e9b99205b0fabb5100569a737d9b82abced184bdacd35" Oct 30 17:45:24.078884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:24.077805 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b"] Oct 30 17:45:24.091513 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:24.091492 4965 scope.go:122] "RemoveContainer" containerID="50a74e40b65174210c0b3d0633914c8d35a40a14e6abd01010f9e68ef665b869" Oct 30 17:45:24.139944 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:24.139908 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf"] Oct 30 17:45:24.163509 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:24.163481 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw"] Oct 30 17:45:24.239908 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:24.239846 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd" Oct 30 17:45:24.291968 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:24.291844 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5-kube-api-access-gg85s\" (UniqueName: \"kubernetes.io/projected/058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5-kube-api-access-gg85s\") pod \"058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5\" (UID: \"058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5\") " Oct 30 17:45:24.291968 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:24.291958 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5-maas-api-service-ca\") pod \"058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5\" (UID: \"058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5\") " Oct 30 17:45:24.292711 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:24.292683 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5-maas-api-service-ca" pod "058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5" (UID: "058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:45:24.294480 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:24.294456 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5-kube-api-access-gg85s" pod "058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5" (UID: "058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5"). InnerVolumeSpecName "kube-api-access-gg85s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:45:24.393711 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:24.393675 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg85s\" (UniqueName: \"kubernetes.io/projected/058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5-kube-api-access-gg85s\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:45:24.393711 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:24.393707 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5-maas-api-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:45:25.125249 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:25.125213 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd" event={"ID":"058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5","Type":"ContainerDied","Data":"a96c3bd9b9c016d5835da8f26d21b72dd1224c03cce282c0f1c6be3c26832cd2"} Oct 30 17:45:25.125747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:25.125726 4965 scope.go:122] "RemoveContainer" containerID="edfad6c40a9b5a8cf8b9542c07c9bf3a43f6f8ee8d9edd011b40a8ab46ee84e8" Oct 30 17:45:25.126401 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:25.125947 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd" Oct 30 17:45:25.134149 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:25.134120 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq" event={"ID":"09351610-5b9d-418c-b902-0e8dcf597ab5","Type":"ContainerStarted","Data":"2f96f983a9fc96cb993b9a55edac2f299e82d85aa8f5a2f2982ccaba0b75ffb4"} Oct 30 17:45:25.371140 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:25.371036 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd"] Oct 30 17:45:25.378416 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:25.378313 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-874b83-29889705-gsndd"] Oct 30 17:45:25.541369 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:25.539368 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf" Oct 30 17:45:25.608704 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:25.608666 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b" Oct 30 17:45:25.631791 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:25.631759 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e5f291f-ff17-41bc-b87f-2beaaa451c13-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e5f291f-ff17-41bc-b87f-2beaaa451c13-maas-api-service-ca\") pod \"2e5f291f-ff17-41bc-b87f-2beaaa451c13\" (UID: \"2e5f291f-ff17-41bc-b87f-2beaaa451c13\") " Oct 30 17:45:25.632120 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:25.631965 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e5f291f-ff17-41bc-b87f-2beaaa451c13-kube-api-access-x5ns9\" (UniqueName: \"kubernetes.io/projected/2e5f291f-ff17-41bc-b87f-2beaaa451c13-kube-api-access-x5ns9\") pod \"2e5f291f-ff17-41bc-b87f-2beaaa451c13\" (UID: \"2e5f291f-ff17-41bc-b87f-2beaaa451c13\") " Oct 30 17:45:25.632222 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:25.632192 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e5f291f-ff17-41bc-b87f-2beaaa451c13-maas-api-service-ca" pod "2e5f291f-ff17-41bc-b87f-2beaaa451c13" (UID: "2e5f291f-ff17-41bc-b87f-2beaaa451c13"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:45:25.632684 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:25.632660 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e5f291f-ff17-41bc-b87f-2beaaa451c13-maas-api-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:45:25.634408 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:25.634383 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e5f291f-ff17-41bc-b87f-2beaaa451c13-kube-api-access-x5ns9" pod "2e5f291f-ff17-41bc-b87f-2beaaa451c13" (UID: "2e5f291f-ff17-41bc-b87f-2beaaa451c13"). InnerVolumeSpecName "kube-api-access-x5ns9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:45:25.638293 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:25.638270 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw" Oct 30 17:45:25.733731 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:25.733693 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e59f243c-037f-4b74-8c60-c7fbbbc10c65-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e59f243c-037f-4b74-8c60-c7fbbbc10c65-maas-api-service-ca\") pod \"e59f243c-037f-4b74-8c60-c7fbbbc10c65\" (UID: \"e59f243c-037f-4b74-8c60-c7fbbbc10c65\") " Oct 30 17:45:25.733982 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:25.733826 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fcbf5b5-34ce-408a-874b-29f6ea910d40-kube-api-access-rgwbv\" (UniqueName: \"kubernetes.io/projected/6fcbf5b5-34ce-408a-874b-29f6ea910d40-kube-api-access-rgwbv\") pod \"6fcbf5b5-34ce-408a-874b-29f6ea910d40\" (UID: \"6fcbf5b5-34ce-408a-874b-29f6ea910d40\") " Oct 30 17:45:25.734091 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:25.733988 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fcbf5b5-34ce-408a-874b-29f6ea910d40-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fcbf5b5-34ce-408a-874b-29f6ea910d40-maas-api-service-ca\") pod \"6fcbf5b5-34ce-408a-874b-29f6ea910d40\" (UID: \"6fcbf5b5-34ce-408a-874b-29f6ea910d40\") " Oct 30 17:45:25.734091 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:25.734073 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e59f243c-037f-4b74-8c60-c7fbbbc10c65-kube-api-access-fqpjv\" (UniqueName: \"kubernetes.io/projected/e59f243c-037f-4b74-8c60-c7fbbbc10c65-kube-api-access-fqpjv\") pod \"e59f243c-037f-4b74-8c60-c7fbbbc10c65\" (UID: \"e59f243c-037f-4b74-8c60-c7fbbbc10c65\") " Oct 30 17:45:25.734307 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:25.734282 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e59f243c-037f-4b74-8c60-c7fbbbc10c65-maas-api-service-ca" pod "e59f243c-037f-4b74-8c60-c7fbbbc10c65" (UID: "e59f243c-037f-4b74-8c60-c7fbbbc10c65"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:45:25.735010 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:25.734977 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fcbf5b5-34ce-408a-874b-29f6ea910d40-maas-api-service-ca" pod "6fcbf5b5-34ce-408a-874b-29f6ea910d40" (UID: "6fcbf5b5-34ce-408a-874b-29f6ea910d40"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:45:25.735108 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:25.734989 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5ns9\" (UniqueName: \"kubernetes.io/projected/2e5f291f-ff17-41bc-b87f-2beaaa451c13-kube-api-access-x5ns9\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:45:25.735108 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:25.735056 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e59f243c-037f-4b74-8c60-c7fbbbc10c65-maas-api-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:45:25.736633 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:25.736600 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fcbf5b5-34ce-408a-874b-29f6ea910d40-kube-api-access-rgwbv" pod "6fcbf5b5-34ce-408a-874b-29f6ea910d40" (UID: "6fcbf5b5-34ce-408a-874b-29f6ea910d40"). InnerVolumeSpecName "kube-api-access-rgwbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:45:25.737101 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:25.737070 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e59f243c-037f-4b74-8c60-c7fbbbc10c65-kube-api-access-fqpjv" pod "e59f243c-037f-4b74-8c60-c7fbbbc10c65" (UID: "e59f243c-037f-4b74-8c60-c7fbbbc10c65"). InnerVolumeSpecName "kube-api-access-fqpjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:45:25.837194 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:25.837156 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgwbv\" (UniqueName: \"kubernetes.io/projected/6fcbf5b5-34ce-408a-874b-29f6ea910d40-kube-api-access-rgwbv\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:45:25.837194 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:25.837189 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fcbf5b5-34ce-408a-874b-29f6ea910d40-maas-api-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:45:25.837194 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:25.837199 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqpjv\" (UniqueName: \"kubernetes.io/projected/e59f243c-037f-4b74-8c60-c7fbbbc10c65-kube-api-access-fqpjv\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:45:26.111377 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:26.111345 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5" path="/var/lib/kubelet/pods/058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5/volumes" Oct 30 17:45:26.147064 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:26.147037 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b" Oct 30 17:45:26.147431 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:26.147035 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b" event={"ID":"6fcbf5b5-34ce-408a-874b-29f6ea910d40","Type":"ContainerDied","Data":"c696fe5c446ac32a3dcb45e9dc56f1745cb80c741a6ac22ec9e37fa6fdf0925e"} Oct 30 17:45:26.147431 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:26.147178 4965 scope.go:122] "RemoveContainer" containerID="ec2ae82be0fc542e6c2b0cf887fe98e6f1ff8103b2bcf13e06e23dc05264fc6e" Oct 30 17:45:26.150508 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:26.150481 4965 generic.go:356] "Generic (PLEG): container finished" podID="09351610-5b9d-418c-b902-0e8dcf597ab5" containerID="2f96f983a9fc96cb993b9a55edac2f299e82d85aa8f5a2f2982ccaba0b75ffb4" exitCode=60 Oct 30 17:45:26.150622 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:26.150549 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq" event={"ID":"09351610-5b9d-418c-b902-0e8dcf597ab5","Type":"ContainerDied","Data":"2f96f983a9fc96cb993b9a55edac2f299e82d85aa8f5a2f2982ccaba0b75ffb4"} Oct 30 17:45:26.150931 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:26.150913 4965 scope.go:122] "RemoveContainer" containerID="2f96f983a9fc96cb993b9a55edac2f299e82d85aa8f5a2f2982ccaba0b75ffb4" Oct 30 17:45:26.151227 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:45:26.151209 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq_odh-ai-gateway-infra(09351610-5b9d-418c-b902-0e8dcf597ab5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq" podUID="09351610-5b9d-418c-b902-0e8dcf597ab5" Oct 30 17:45:26.152222 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:26.152203 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw" Oct 30 17:45:26.152316 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:26.152276 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw" event={"ID":"e59f243c-037f-4b74-8c60-c7fbbbc10c65","Type":"ContainerDied","Data":"a61a024f8ac2f0ed7b5cf6812f66f9f25877407fa48ab4078cee60c68532ec28"} Oct 30 17:45:26.155152 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:26.155128 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf" event={"ID":"2e5f291f-ff17-41bc-b87f-2beaaa451c13","Type":"ContainerDied","Data":"2dfd728d3309c4b9783ff984cfbf7776c51575a6a0f2f6c1ffd739cf07b549d2"} Oct 30 17:45:26.155250 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:26.155164 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf" Oct 30 17:45:26.159940 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:26.159922 4965 scope.go:122] "RemoveContainer" containerID="50a74e40b65174210c0b3d0633914c8d35a40a14e6abd01010f9e68ef665b869" Oct 30 17:45:26.174895 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:26.174833 4965 scope.go:122] "RemoveContainer" containerID="0f49d4fd10c6b1c82dc457606bf3b8006cb8ca5f8a971e6990a5a073bf199d7c" Oct 30 17:45:26.185102 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:26.185080 4965 scope.go:122] "RemoveContainer" containerID="a786716774b9cf3114dc6475ac88732a61d462d7bd0cd1ee5e96bd8389b56f46" Oct 30 17:45:26.263204 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:26.258818 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b"] Oct 30 17:45:26.270888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:26.266337 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fb0d-29889705-n258b"] Oct 30 17:45:26.291166 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:26.291135 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf"] Oct 30 17:45:26.295980 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:26.295958 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889705-cwzhf"] Oct 30 17:45:26.299819 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:26.299799 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq"] Oct 30 17:45:26.331237 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:26.331203 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw"] Oct 30 17:45:26.341203 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:26.341167 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889705-m6vhw"] Oct 30 17:45:27.340951 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:27.340925 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq" Oct 30 17:45:27.385591 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:27.385560 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09351610-5b9d-418c-b902-0e8dcf597ab5-kube-api-access-4c6bj\" (UniqueName: \"kubernetes.io/projected/09351610-5b9d-418c-b902-0e8dcf597ab5-kube-api-access-4c6bj\") pod \"09351610-5b9d-418c-b902-0e8dcf597ab5\" (UID: \"09351610-5b9d-418c-b902-0e8dcf597ab5\") " Oct 30 17:45:27.385760 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:27.385736 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09351610-5b9d-418c-b902-0e8dcf597ab5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09351610-5b9d-418c-b902-0e8dcf597ab5-maas-api-service-ca\") pod \"09351610-5b9d-418c-b902-0e8dcf597ab5\" (UID: \"09351610-5b9d-418c-b902-0e8dcf597ab5\") " Oct 30 17:45:27.386186 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:27.386134 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09351610-5b9d-418c-b902-0e8dcf597ab5-maas-api-service-ca" pod "09351610-5b9d-418c-b902-0e8dcf597ab5" (UID: "09351610-5b9d-418c-b902-0e8dcf597ab5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:45:27.386699 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:27.386672 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09351610-5b9d-418c-b902-0e8dcf597ab5-maas-api-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:45:27.388887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:27.388186 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09351610-5b9d-418c-b902-0e8dcf597ab5-kube-api-access-4c6bj" pod "09351610-5b9d-418c-b902-0e8dcf597ab5" (UID: "09351610-5b9d-418c-b902-0e8dcf597ab5"). InnerVolumeSpecName "kube-api-access-4c6bj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:45:27.488017 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:27.487941 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c6bj\" (UniqueName: \"kubernetes.io/projected/09351610-5b9d-418c-b902-0e8dcf597ab5-kube-api-access-4c6bj\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:45:28.102564 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:28.102529 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e5f291f-ff17-41bc-b87f-2beaaa451c13" path="/var/lib/kubelet/pods/2e5f291f-ff17-41bc-b87f-2beaaa451c13/volumes" Oct 30 17:45:28.103090 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:28.103073 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fcbf5b5-34ce-408a-874b-29f6ea910d40" path="/var/lib/kubelet/pods/6fcbf5b5-34ce-408a-874b-29f6ea910d40/volumes" Oct 30 17:45:28.103486 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:28.103474 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e59f243c-037f-4b74-8c60-c7fbbbc10c65" path="/var/lib/kubelet/pods/e59f243c-037f-4b74-8c60-c7fbbbc10c65/volumes" Oct 30 17:45:28.176555 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:28.176521 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq" Oct 30 17:45:28.176555 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:28.176544 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq" event={"ID":"09351610-5b9d-418c-b902-0e8dcf597ab5","Type":"ContainerDied","Data":"001da2a36d697a37e8ea23d2ccedbc4658db95d6d17e531148376fe225e4e26a"} Oct 30 17:45:28.176778 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:28.176603 4965 scope.go:122] "RemoveContainer" containerID="2f96f983a9fc96cb993b9a55edac2f299e82d85aa8f5a2f2982ccaba0b75ffb4" Oct 30 17:45:28.231899 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:28.231830 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq"] Oct 30 17:45:28.238499 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:28.238474 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f8b48d-29889705-t2nsq"] Oct 30 17:45:30.130779 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:30.130744 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09351610-5b9d-418c-b902-0e8dcf597ab5" path="/var/lib/kubelet/pods/09351610-5b9d-418c-b902-0e8dcf597ab5/volumes" Oct 30 17:45:36.006822 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:45:36.006787 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:45:41.166092 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:41.166048 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6"] Oct 30 17:45:41.166604 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:41.166355 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" podUID="a628b339-91ea-46d8-9935-fc9174b9e4d2" containerName="istio-proxy" containerID="cri-o://d2ab40089b0bd25196e93a3fe8fd114e640acd3fbcb87154876db0f62ac9d92f" gracePeriod=30 Oct 30 17:45:43.302068 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.302025 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs"] Oct 30 17:45:43.302962 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.302943 4965 cpu_manager.go:414] "containerMap: removing container" podUID="058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5" containerName="cleanup" Oct 30 17:45:43.302962 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.302962 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5" containerName="cleanup" Oct 30 17:45:43.303115 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.302974 4965 cpu_manager.go:414] "containerMap: removing container" podUID="6fcbf5b5-34ce-408a-874b-29f6ea910d40" containerName="cleanup" Oct 30 17:45:43.303115 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.302979 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fcbf5b5-34ce-408a-874b-29f6ea910d40" containerName="cleanup" Oct 30 17:45:43.303115 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.302990 4965 cpu_manager.go:414] "containerMap: removing container" podUID="09351610-5b9d-418c-b902-0e8dcf597ab5" containerName="cleanup" Oct 30 17:45:43.303115 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.302997 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09351610-5b9d-418c-b902-0e8dcf597ab5" containerName="cleanup" Oct 30 17:45:43.303115 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.303017 4965 cpu_manager.go:414] "containerMap: removing container" podUID="09351610-5b9d-418c-b902-0e8dcf597ab5" containerName="cleanup" Oct 30 17:45:43.303115 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.303022 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09351610-5b9d-418c-b902-0e8dcf597ab5" containerName="cleanup" Oct 30 17:45:43.303115 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.303032 4965 cpu_manager.go:414] "containerMap: removing container" podUID="2e5f291f-ff17-41bc-b87f-2beaaa451c13" containerName="cleanup" Oct 30 17:45:43.303115 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.303038 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e5f291f-ff17-41bc-b87f-2beaaa451c13" containerName="cleanup" Oct 30 17:45:43.303115 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.303053 4965 cpu_manager.go:414] "containerMap: removing container" podUID="e59f243c-037f-4b74-8c60-c7fbbbc10c65" containerName="cleanup" Oct 30 17:45:43.303115 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.303058 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e59f243c-037f-4b74-8c60-c7fbbbc10c65" containerName="cleanup" Oct 30 17:45:43.303115 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.303077 4965 cpu_manager.go:414] "containerMap: removing container" podUID="058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5" containerName="cleanup" Oct 30 17:45:43.303115 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.303082 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5" containerName="cleanup" Oct 30 17:45:43.303115 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.303089 4965 cpu_manager.go:414] "containerMap: removing container" podUID="2e5f291f-ff17-41bc-b87f-2beaaa451c13" containerName="cleanup" Oct 30 17:45:43.303115 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.303095 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e5f291f-ff17-41bc-b87f-2beaaa451c13" containerName="cleanup" Oct 30 17:45:43.303115 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.303106 4965 cpu_manager.go:414] "containerMap: removing container" podUID="2e5f291f-ff17-41bc-b87f-2beaaa451c13" containerName="cleanup" Oct 30 17:45:43.303115 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.303112 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e5f291f-ff17-41bc-b87f-2beaaa451c13" containerName="cleanup" Oct 30 17:45:43.303115 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.303120 4965 cpu_manager.go:414] "containerMap: removing container" podUID="058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5" containerName="cleanup" Oct 30 17:45:43.303115 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.303124 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5" containerName="cleanup" Oct 30 17:45:43.303936 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.303132 4965 cpu_manager.go:414] "containerMap: removing container" podUID="6fcbf5b5-34ce-408a-874b-29f6ea910d40" containerName="cleanup" Oct 30 17:45:43.303936 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.303136 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fcbf5b5-34ce-408a-874b-29f6ea910d40" containerName="cleanup" Oct 30 17:45:43.303936 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.303160 4965 cpu_manager.go:414] "containerMap: removing container" podUID="e59f243c-037f-4b74-8c60-c7fbbbc10c65" containerName="cleanup" Oct 30 17:45:43.303936 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.303164 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e59f243c-037f-4b74-8c60-c7fbbbc10c65" containerName="cleanup" Oct 30 17:45:43.303936 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.303299 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5" containerName="cleanup" Oct 30 17:45:43.303936 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.303308 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e5f291f-ff17-41bc-b87f-2beaaa451c13" containerName="cleanup" Oct 30 17:45:43.303936 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.303314 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fcbf5b5-34ce-408a-874b-29f6ea910d40" containerName="cleanup" Oct 30 17:45:43.303936 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.303320 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e5f291f-ff17-41bc-b87f-2beaaa451c13" containerName="cleanup" Oct 30 17:45:43.303936 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.303328 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5" containerName="cleanup" Oct 30 17:45:43.303936 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.303334 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fcbf5b5-34ce-408a-874b-29f6ea910d40" containerName="cleanup" Oct 30 17:45:43.303936 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.303341 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="e59f243c-037f-4b74-8c60-c7fbbbc10c65" containerName="cleanup" Oct 30 17:45:43.303936 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.303347 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="e59f243c-037f-4b74-8c60-c7fbbbc10c65" containerName="cleanup" Oct 30 17:45:43.303936 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.303355 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="09351610-5b9d-418c-b902-0e8dcf597ab5" containerName="cleanup" Oct 30 17:45:43.303936 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.303360 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="09351610-5b9d-418c-b902-0e8dcf597ab5" containerName="cleanup" Oct 30 17:45:43.303936 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.303367 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="09351610-5b9d-418c-b902-0e8dcf597ab5" containerName="cleanup" Oct 30 17:45:43.312153 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.312128 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:45:43.340645 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.340610 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs"] Oct 30 17:45:43.341840 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.341809 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-istio-envoy\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:45:43.342159 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.342140 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-workload-certs\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:45:43.342314 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.342198 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqgpl\" (UniqueName: \"kubernetes.io/projected/81aec189-33b7-48e3-a558-65938570fc4f-kube-api-access-pqgpl\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:45:43.342314 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.342257 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-credential-socket\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:45:43.342487 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.342380 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81aec189-33b7-48e3-a558-65938570fc4f-istio-token\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:45:43.342487 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.342465 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-workload-socket\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:45:43.342653 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.342520 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81aec189-33b7-48e3-a558-65938570fc4f-istiod-ca-cert\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:45:43.342653 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.342581 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-istio-data\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:45:43.342821 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.342659 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81aec189-33b7-48e3-a558-65938570fc4f-istio-podinfo\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:45:43.405711 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.405677 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-bba90b-openshift-default-dockercfg-d9vst\"" Oct 30 17:45:43.444385 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.444350 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-istio-envoy\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:45:43.444567 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.444481 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-workload-certs\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:45:43.444567 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.444514 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqgpl\" (UniqueName: \"kubernetes.io/projected/81aec189-33b7-48e3-a558-65938570fc4f-kube-api-access-pqgpl\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:45:43.444567 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.444539 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-credential-socket\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:45:43.444729 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.444642 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81aec189-33b7-48e3-a558-65938570fc4f-istio-token\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:45:43.444729 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.444711 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-workload-socket\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:45:43.444839 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.444752 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81aec189-33b7-48e3-a558-65938570fc4f-istiod-ca-cert\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:45:43.444839 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.444803 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-istio-data\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:45:43.444961 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.444890 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81aec189-33b7-48e3-a558-65938570fc4f-istio-podinfo\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:45:43.444961 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.444909 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-workload-certs\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:45:43.444961 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.444937 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-credential-socket\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:45:43.445235 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.445212 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-workload-socket\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:45:43.445349 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.445330 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-istio-data\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:45:43.445549 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.445530 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81aec189-33b7-48e3-a558-65938570fc4f-istiod-ca-cert\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:45:43.447406 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.447379 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-istio-envoy\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:45:43.447516 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.447506 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81aec189-33b7-48e3-a558-65938570fc4f-istio-podinfo\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:45:43.467600 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.467577 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81aec189-33b7-48e3-a558-65938570fc4f-istio-token\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:45:43.467939 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.467916 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqgpl\" (UniqueName: \"kubernetes.io/projected/81aec189-33b7-48e3-a558-65938570fc4f-kube-api-access-pqgpl\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:45:43.640399 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.640364 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:45:43.963683 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.963650 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs"] Oct 30 17:45:43.963847 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:45:43.963824 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81aec189_33b7_48e3_a558_65938570fc4f.slice/crio-58ba539b596ab93d066a57be2720edc877112a5e292b43129f696c4a2d606133 WatchSource:0}: Error finding container 58ba539b596ab93d066a57be2720edc877112a5e292b43129f696c4a2d606133: Status 404 returned error can't find the container with id 58ba539b596ab93d066a57be2720edc877112a5e292b43129f696c4a2d606133 Oct 30 17:45:43.966610 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.966575 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:45:43.966723 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.966655 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:45:43.966723 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:43.966701 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:45:44.362941 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:44.362903 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" event={"ID":"81aec189-33b7-48e3-a558-65938570fc4f","Type":"ContainerStarted","Data":"fd97f08d23fdd5ee2c8004b0f9ffe3adc0783bf540d2bb65c24bd0f471175a6b"} Oct 30 17:45:44.363349 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:44.362972 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" event={"ID":"81aec189-33b7-48e3-a558-65938570fc4f","Type":"ContainerStarted","Data":"58ba539b596ab93d066a57be2720edc877112a5e292b43129f696c4a2d606133"} Oct 30 17:45:44.434504 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:44.434447 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" podStartSLOduration=1.43443261 podStartE2EDuration="1.43443261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:44.419057522 +0000 UTC m=+2002.471157474" watchObservedRunningTime="2026-10-30 17:45:44.43443261 +0000 UTC m=+2002.486532596" Oct 30 17:45:44.641462 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:44.641381 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:45:45.642893 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:45.642842 4965 patch_prober.go:29] interesting pod/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 17:45:45.643373 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:45.642918 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" podUID="81aec189-33b7-48e3-a558-65938570fc4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 17:45:46.512062 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:46.511979 4965 generic.go:356] "Generic (PLEG): container finished" podID="a628b339-91ea-46d8-9935-fc9174b9e4d2" containerID="d2ab40089b0bd25196e93a3fe8fd114e640acd3fbcb87154876db0f62ac9d92f" exitCode=0 Oct 30 17:45:46.513343 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:46.513306 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" event={"ID":"a628b339-91ea-46d8-9935-fc9174b9e4d2","Type":"ContainerDied","Data":"d2ab40089b0bd25196e93a3fe8fd114e640acd3fbcb87154876db0f62ac9d92f"} Oct 30 17:45:46.631341 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:46.631308 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:45:46.641401 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:46.641368 4965 patch_prober.go:29] interesting pod/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 17:45:46.641545 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:46.641419 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" podUID="81aec189-33b7-48e3-a558-65938570fc4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 17:45:46.725198 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:46.725156 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-workload-socket\") pod \"a628b339-91ea-46d8-9935-fc9174b9e4d2\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " Oct 30 17:45:46.725672 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:46.725302 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-credential-socket\") pod \"a628b339-91ea-46d8-9935-fc9174b9e4d2\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " Oct 30 17:45:46.725672 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:46.725390 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a628b339-91ea-46d8-9935-fc9174b9e4d2-kube-api-access-dmlxt\" (UniqueName: \"kubernetes.io/projected/a628b339-91ea-46d8-9935-fc9174b9e4d2-kube-api-access-dmlxt\") pod \"a628b339-91ea-46d8-9935-fc9174b9e4d2\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " Oct 30 17:45:46.725672 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:46.725425 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-workload-socket" pod "a628b339-91ea-46d8-9935-fc9174b9e4d2" (UID: "a628b339-91ea-46d8-9935-fc9174b9e4d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:45:46.725672 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:46.725502 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-istio-data\") pod \"a628b339-91ea-46d8-9935-fc9174b9e4d2\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " Oct 30 17:45:46.725672 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:46.725585 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a628b339-91ea-46d8-9935-fc9174b9e4d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a628b339-91ea-46d8-9935-fc9174b9e4d2-istiod-ca-cert\") pod \"a628b339-91ea-46d8-9935-fc9174b9e4d2\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " Oct 30 17:45:46.725995 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:46.725675 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a628b339-91ea-46d8-9935-fc9174b9e4d2-istio-token\" (UniqueName: \"kubernetes.io/projected/a628b339-91ea-46d8-9935-fc9174b9e4d2-istio-token\") pod \"a628b339-91ea-46d8-9935-fc9174b9e4d2\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " Oct 30 17:45:46.725995 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:46.725769 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a628b339-91ea-46d8-9935-fc9174b9e4d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a628b339-91ea-46d8-9935-fc9174b9e4d2-istio-podinfo\") pod \"a628b339-91ea-46d8-9935-fc9174b9e4d2\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " Oct 30 17:45:46.725995 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:46.725857 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-istio-envoy\") pod \"a628b339-91ea-46d8-9935-fc9174b9e4d2\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " Oct 30 17:45:46.726171 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:46.726020 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-workload-certs\") pod \"a628b339-91ea-46d8-9935-fc9174b9e4d2\" (UID: \"a628b339-91ea-46d8-9935-fc9174b9e4d2\") " Oct 30 17:45:46.727350 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:46.727320 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:45:46.727567 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:46.727544 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-workload-certs" pod "a628b339-91ea-46d8-9935-fc9174b9e4d2" (UID: "a628b339-91ea-46d8-9935-fc9174b9e4d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:45:46.727758 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:46.727737 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-istio-data" pod "a628b339-91ea-46d8-9935-fc9174b9e4d2" (UID: "a628b339-91ea-46d8-9935-fc9174b9e4d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:45:46.729107 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:46.728056 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a628b339-91ea-46d8-9935-fc9174b9e4d2-istiod-ca-cert" pod "a628b339-91ea-46d8-9935-fc9174b9e4d2" (UID: "a628b339-91ea-46d8-9935-fc9174b9e4d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:45:46.732556 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:46.732277 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-credential-socket" pod "a628b339-91ea-46d8-9935-fc9174b9e4d2" (UID: "a628b339-91ea-46d8-9935-fc9174b9e4d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:45:46.735368 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:46.734899 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-istio-envoy" pod "a628b339-91ea-46d8-9935-fc9174b9e4d2" (UID: "a628b339-91ea-46d8-9935-fc9174b9e4d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:45:46.742463 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:46.742432 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a628b339-91ea-46d8-9935-fc9174b9e4d2-kube-api-access-dmlxt" pod "a628b339-91ea-46d8-9935-fc9174b9e4d2" (UID: "a628b339-91ea-46d8-9935-fc9174b9e4d2"). InnerVolumeSpecName "kube-api-access-dmlxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:45:46.744632 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:46.744034 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a628b339-91ea-46d8-9935-fc9174b9e4d2-istio-podinfo" pod "a628b339-91ea-46d8-9935-fc9174b9e4d2" (UID: "a628b339-91ea-46d8-9935-fc9174b9e4d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:45:46.745489 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:46.745463 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a628b339-91ea-46d8-9935-fc9174b9e4d2-istio-token" pod "a628b339-91ea-46d8-9935-fc9174b9e4d2" (UID: "a628b339-91ea-46d8-9935-fc9174b9e4d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:45:46.829424 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:46.829390 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:45:46.829424 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:46.829422 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmlxt\" (UniqueName: \"kubernetes.io/projected/a628b339-91ea-46d8-9935-fc9174b9e4d2-kube-api-access-dmlxt\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:45:46.829424 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:46.829432 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:45:46.829680 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:46.829441 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a628b339-91ea-46d8-9935-fc9174b9e4d2-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:45:46.829680 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:46.829450 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a628b339-91ea-46d8-9935-fc9174b9e4d2-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:45:46.829680 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:46.829458 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a628b339-91ea-46d8-9935-fc9174b9e4d2-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:45:46.829680 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:46.829465 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:45:46.829680 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:46.829473 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a628b339-91ea-46d8-9935-fc9174b9e4d2-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:45:47.521655 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:47.521622 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" Oct 30 17:45:47.521846 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:47.521625 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6" event={"ID":"a628b339-91ea-46d8-9935-fc9174b9e4d2","Type":"ContainerDied","Data":"3780e05465ee8b08e3441f2c85a70a842ac7039dff370211bfe3e59807423e11"} Oct 30 17:45:47.521846 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:47.521760 4965 scope.go:122] "RemoveContainer" containerID="d2ab40089b0bd25196e93a3fe8fd114e640acd3fbcb87154876db0f62ac9d92f" Oct 30 17:45:47.600187 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:47.600150 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6"] Oct 30 17:45:47.612014 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:47.611985 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-nj5w6"] Oct 30 17:45:47.640876 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:47.640840 4965 patch_prober.go:29] interesting pod/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 17:45:47.641003 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:47.640905 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" podUID="81aec189-33b7-48e3-a558-65938570fc4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 17:45:48.101215 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:48.101179 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a628b339-91ea-46d8-9935-fc9174b9e4d2" path="/var/lib/kubelet/pods/a628b339-91ea-46d8-9935-fc9174b9e4d2/volumes" Oct 30 17:45:48.641014 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:48.640981 4965 patch_prober.go:29] interesting pod/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 17:45:48.641221 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:48.641037 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" podUID="81aec189-33b7-48e3-a558-65938570fc4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 17:45:49.654958 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:49.648310 4965 patch_prober.go:29] interesting pod/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 17:45:49.654958 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:49.648372 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" podUID="81aec189-33b7-48e3-a558-65938570fc4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 17:45:50.641162 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:50.641127 4965 patch_prober.go:29] interesting pod/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 17:45:50.641362 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:50.641182 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" podUID="81aec189-33b7-48e3-a558-65938570fc4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 17:45:51.986599 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:51.986562 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:45:51.987169 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:51.987045 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:45:51.988252 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:45:51.988227 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:46:02.348453 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:02.348409 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z"] Oct 30 17:46:02.349845 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:02.349801 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" podUID="74e7c579-cc3b-4bd5-998e-3bc0a50aed85" containerName="istio-proxy" containerID="cri-o://26c5e7f5f5ca6f56d9e6c998a30f9b49c31a7199766784fc813dda2f5309ed45" gracePeriod=30 Oct 30 17:46:02.381912 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:02.379662 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m"] Oct 30 17:46:02.382210 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:02.382176 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" podUID="1114e4ad-fc83-4876-9ffe-c47afce03fc5" containerName="istio-proxy" containerID="cri-o://979d62e7a612b5159e8d58b445165dd65de5af122cce7c42216d87e0d8c8f022" gracePeriod=30 Oct 30 17:46:05.998328 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:46:05.998238 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:46:07.685429 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.685401 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:46:07.735807 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.735764 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istio-podinfo\") pod \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " Oct 30 17:46:07.735993 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.735878 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-workload-certs\") pod \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " Oct 30 17:46:07.735993 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.735977 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-workload-socket\") pod \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " Oct 30 17:46:07.736127 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.736015 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istio-token\" (UniqueName: \"kubernetes.io/projected/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istio-token\") pod \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " Oct 30 17:46:07.736269 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.736206 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-workload-certs" pod "74e7c579-cc3b-4bd5-998e-3bc0a50aed85" (UID: "74e7c579-cc3b-4bd5-998e-3bc0a50aed85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:46:07.736409 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.736274 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-workload-socket" pod "74e7c579-cc3b-4bd5-998e-3bc0a50aed85" (UID: "74e7c579-cc3b-4bd5-998e-3bc0a50aed85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:46:07.736409 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.736356 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-kube-api-access-mkhv7\" (UniqueName: \"kubernetes.io/projected/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-kube-api-access-mkhv7\") pod \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " Oct 30 17:46:07.736527 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.736408 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istio-envoy\") pod \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " Oct 30 17:46:07.736527 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.736442 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-credential-socket\") pod \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " Oct 30 17:46:07.736527 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.736521 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istio-data\") pod \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " Oct 30 17:46:07.736691 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.736572 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istiod-ca-cert\") pod \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\" (UID: \"74e7c579-cc3b-4bd5-998e-3bc0a50aed85\") " Oct 30 17:46:07.737256 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.737065 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istio-data" pod "74e7c579-cc3b-4bd5-998e-3bc0a50aed85" (UID: "74e7c579-cc3b-4bd5-998e-3bc0a50aed85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:46:07.737370 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.737355 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istiod-ca-cert" pod "74e7c579-cc3b-4bd5-998e-3bc0a50aed85" (UID: "74e7c579-cc3b-4bd5-998e-3bc0a50aed85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:46:07.737509 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.737450 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:46:07.737509 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.737475 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:46:07.737509 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.737488 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:46:07.737874 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.737822 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-credential-socket" pod "74e7c579-cc3b-4bd5-998e-3bc0a50aed85" (UID: "74e7c579-cc3b-4bd5-998e-3bc0a50aed85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:46:07.738533 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.738506 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istio-podinfo" pod "74e7c579-cc3b-4bd5-998e-3bc0a50aed85" (UID: "74e7c579-cc3b-4bd5-998e-3bc0a50aed85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:46:07.738677 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.738655 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istio-token" pod "74e7c579-cc3b-4bd5-998e-3bc0a50aed85" (UID: "74e7c579-cc3b-4bd5-998e-3bc0a50aed85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:46:07.739265 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.739242 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-kube-api-access-mkhv7" pod "74e7c579-cc3b-4bd5-998e-3bc0a50aed85" (UID: "74e7c579-cc3b-4bd5-998e-3bc0a50aed85"). InnerVolumeSpecName "kube-api-access-mkhv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:46:07.739568 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.739550 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istio-envoy" pod "74e7c579-cc3b-4bd5-998e-3bc0a50aed85" (UID: "74e7c579-cc3b-4bd5-998e-3bc0a50aed85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:46:07.767391 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.765107 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:46:07.838222 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.838186 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istio-token\" (UniqueName: \"kubernetes.io/projected/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istio-token\") pod \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " Oct 30 17:46:07.838406 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.838283 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1114e4ad-fc83-4876-9ffe-c47afce03fc5-kube-api-access-csbkw\" (UniqueName: \"kubernetes.io/projected/1114e4ad-fc83-4876-9ffe-c47afce03fc5-kube-api-access-csbkw\") pod \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " Oct 30 17:46:07.838537 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.838518 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istio-data\") pod \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " Oct 30 17:46:07.838646 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.838586 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istio-podinfo\") pod \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " Oct 30 17:46:07.838646 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.838628 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istiod-ca-cert\") pod \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " Oct 30 17:46:07.838770 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.838657 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-workload-certs\") pod \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " Oct 30 17:46:07.838770 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.838715 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-workload-socket\") pod \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " Oct 30 17:46:07.838914 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.838839 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istio-data" pod "1114e4ad-fc83-4876-9ffe-c47afce03fc5" (UID: "1114e4ad-fc83-4876-9ffe-c47afce03fc5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:46:07.838914 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.838858 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istio-envoy\") pod \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " Oct 30 17:46:07.839036 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.838918 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-credential-socket\") pod \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\" (UID: \"1114e4ad-fc83-4876-9ffe-c47afce03fc5\") " Oct 30 17:46:07.839279 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.839162 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istiod-ca-cert" pod "1114e4ad-fc83-4876-9ffe-c47afce03fc5" (UID: "1114e4ad-fc83-4876-9ffe-c47afce03fc5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:46:07.839948 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.839521 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-workload-certs" pod "1114e4ad-fc83-4876-9ffe-c47afce03fc5" (UID: "1114e4ad-fc83-4876-9ffe-c47afce03fc5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:46:07.839948 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.839720 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-credential-socket" pod "1114e4ad-fc83-4876-9ffe-c47afce03fc5" (UID: "1114e4ad-fc83-4876-9ffe-c47afce03fc5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:46:07.839948 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.839778 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:46:07.839948 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.839801 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:46:07.839948 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.839801 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-workload-socket" pod "1114e4ad-fc83-4876-9ffe-c47afce03fc5" (UID: "1114e4ad-fc83-4876-9ffe-c47afce03fc5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:46:07.839948 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.839814 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:46:07.839948 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.839847 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:46:07.839948 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.839908 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:46:07.839948 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.839925 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:46:07.839948 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.839939 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkhv7\" (UniqueName: \"kubernetes.io/projected/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-kube-api-access-mkhv7\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:46:07.840555 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.839959 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:46:07.840555 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.839973 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e7c579-cc3b-4bd5-998e-3bc0a50aed85-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:46:07.841349 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.841327 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1114e4ad-fc83-4876-9ffe-c47afce03fc5-kube-api-access-csbkw" pod "1114e4ad-fc83-4876-9ffe-c47afce03fc5" (UID: "1114e4ad-fc83-4876-9ffe-c47afce03fc5"). InnerVolumeSpecName "kube-api-access-csbkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:46:07.842908 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.842535 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istio-envoy" pod "1114e4ad-fc83-4876-9ffe-c47afce03fc5" (UID: "1114e4ad-fc83-4876-9ffe-c47afce03fc5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:46:07.843102 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.843074 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istio-podinfo" pod "1114e4ad-fc83-4876-9ffe-c47afce03fc5" (UID: "1114e4ad-fc83-4876-9ffe-c47afce03fc5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:46:07.843309 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.843276 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istio-token" pod "1114e4ad-fc83-4876-9ffe-c47afce03fc5" (UID: "1114e4ad-fc83-4876-9ffe-c47afce03fc5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:46:07.941966 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.941891 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:46:07.941966 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.941918 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:46:07.941966 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.941927 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:46:07.941966 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.941935 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1114e4ad-fc83-4876-9ffe-c47afce03fc5-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:46:07.941966 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.941946 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1114e4ad-fc83-4876-9ffe-c47afce03fc5-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:46:07.941966 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:07.941957 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csbkw\" (UniqueName: \"kubernetes.io/projected/1114e4ad-fc83-4876-9ffe-c47afce03fc5-kube-api-access-csbkw\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:46:08.305907 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:08.305851 4965 generic.go:356] "Generic (PLEG): container finished" podID="1114e4ad-fc83-4876-9ffe-c47afce03fc5" containerID="979d62e7a612b5159e8d58b445165dd65de5af122cce7c42216d87e0d8c8f022" exitCode=0 Oct 30 17:46:08.305907 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:08.305888 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" event={"ID":"1114e4ad-fc83-4876-9ffe-c47afce03fc5","Type":"ContainerDied","Data":"979d62e7a612b5159e8d58b445165dd65de5af122cce7c42216d87e0d8c8f022"} Oct 30 17:46:08.306167 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:08.305931 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" Oct 30 17:46:08.306167 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:08.305954 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m" event={"ID":"1114e4ad-fc83-4876-9ffe-c47afce03fc5","Type":"ContainerDied","Data":"ecd40e50b6b884a8e489252bc8dd38ad38414465a526858756cef9925a39ee13"} Oct 30 17:46:08.306167 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:08.305986 4965 scope.go:122] "RemoveContainer" containerID="979d62e7a612b5159e8d58b445165dd65de5af122cce7c42216d87e0d8c8f022" Oct 30 17:46:08.307818 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:08.307792 4965 generic.go:356] "Generic (PLEG): container finished" podID="74e7c579-cc3b-4bd5-998e-3bc0a50aed85" containerID="26c5e7f5f5ca6f56d9e6c998a30f9b49c31a7199766784fc813dda2f5309ed45" exitCode=0 Oct 30 17:46:08.307964 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:08.307841 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" Oct 30 17:46:08.307964 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:08.307892 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" event={"ID":"74e7c579-cc3b-4bd5-998e-3bc0a50aed85","Type":"ContainerDied","Data":"26c5e7f5f5ca6f56d9e6c998a30f9b49c31a7199766784fc813dda2f5309ed45"} Oct 30 17:46:08.307964 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:08.307947 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z" event={"ID":"74e7c579-cc3b-4bd5-998e-3bc0a50aed85","Type":"ContainerDied","Data":"d635d346c042f300a9dd0e5eab7d279b6e45665a0538e0ed87efcfca5d9ccc39"} Oct 30 17:46:08.319377 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:08.319345 4965 scope.go:122] "RemoveContainer" containerID="979d62e7a612b5159e8d58b445165dd65de5af122cce7c42216d87e0d8c8f022" Oct 30 17:46:08.319666 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:46:08.319645 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"979d62e7a612b5159e8d58b445165dd65de5af122cce7c42216d87e0d8c8f022\": container with ID starting with 979d62e7a612b5159e8d58b445165dd65de5af122cce7c42216d87e0d8c8f022 not found: ID does not exist" containerID="979d62e7a612b5159e8d58b445165dd65de5af122cce7c42216d87e0d8c8f022" Oct 30 17:46:08.319726 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:08.319675 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"979d62e7a612b5159e8d58b445165dd65de5af122cce7c42216d87e0d8c8f022"} err="failed to get container status \"979d62e7a612b5159e8d58b445165dd65de5af122cce7c42216d87e0d8c8f022\": rpc error: code = NotFound desc = could not find container \"979d62e7a612b5159e8d58b445165dd65de5af122cce7c42216d87e0d8c8f022\": container with ID starting with 979d62e7a612b5159e8d58b445165dd65de5af122cce7c42216d87e0d8c8f022 not found: ID does not exist" Oct 30 17:46:08.319726 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:08.319691 4965 scope.go:122] "RemoveContainer" containerID="26c5e7f5f5ca6f56d9e6c998a30f9b49c31a7199766784fc813dda2f5309ed45" Oct 30 17:46:08.334385 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:08.334365 4965 scope.go:122] "RemoveContainer" containerID="26c5e7f5f5ca6f56d9e6c998a30f9b49c31a7199766784fc813dda2f5309ed45" Oct 30 17:46:08.334754 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:46:08.334730 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26c5e7f5f5ca6f56d9e6c998a30f9b49c31a7199766784fc813dda2f5309ed45\": container with ID starting with 26c5e7f5f5ca6f56d9e6c998a30f9b49c31a7199766784fc813dda2f5309ed45 not found: ID does not exist" containerID="26c5e7f5f5ca6f56d9e6c998a30f9b49c31a7199766784fc813dda2f5309ed45" Oct 30 17:46:08.334910 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:08.334759 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26c5e7f5f5ca6f56d9e6c998a30f9b49c31a7199766784fc813dda2f5309ed45"} err="failed to get container status \"26c5e7f5f5ca6f56d9e6c998a30f9b49c31a7199766784fc813dda2f5309ed45\": rpc error: code = NotFound desc = could not find container \"26c5e7f5f5ca6f56d9e6c998a30f9b49c31a7199766784fc813dda2f5309ed45\": container with ID starting with 26c5e7f5f5ca6f56d9e6c998a30f9b49c31a7199766784fc813dda2f5309ed45 not found: ID does not exist" Oct 30 17:46:08.368855 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:08.368826 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m"] Oct 30 17:46:08.375080 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:08.375057 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-2bc9m"] Oct 30 17:46:08.413725 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:08.413681 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z"] Oct 30 17:46:08.425940 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:08.425912 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-ktz8z"] Oct 30 17:46:10.291928 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:10.290672 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1114e4ad-fc83-4876-9ffe-c47afce03fc5" path="/var/lib/kubelet/pods/1114e4ad-fc83-4876-9ffe-c47afce03fc5/volumes" Oct 30 17:46:10.291928 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:10.291377 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74e7c579-cc3b-4bd5-998e-3bc0a50aed85" path="/var/lib/kubelet/pods/74e7c579-cc3b-4bd5-998e-3bc0a50aed85/volumes" Oct 30 17:46:18.813743 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:18.813706 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-303ba5-7bfc9db6c8-shn24"] Oct 30 17:46:18.814567 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:18.814550 4965 cpu_manager.go:414] "containerMap: removing container" podUID="6fcbf5b5-34ce-408a-874b-29f6ea910d40" containerName="cleanup" Oct 30 17:46:18.814622 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:18.814569 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fcbf5b5-34ce-408a-874b-29f6ea910d40" containerName="cleanup" Oct 30 17:46:18.814622 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:18.814586 4965 cpu_manager.go:414] "containerMap: removing container" podUID="1114e4ad-fc83-4876-9ffe-c47afce03fc5" containerName="istio-proxy" Oct 30 17:46:18.814622 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:18.814591 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1114e4ad-fc83-4876-9ffe-c47afce03fc5" containerName="istio-proxy" Oct 30 17:46:18.814622 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:18.814604 4965 cpu_manager.go:414] "containerMap: removing container" podUID="a628b339-91ea-46d8-9935-fc9174b9e4d2" containerName="istio-proxy" Oct 30 17:46:18.814622 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:18.814609 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a628b339-91ea-46d8-9935-fc9174b9e4d2" containerName="istio-proxy" Oct 30 17:46:18.814622 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:18.814622 4965 cpu_manager.go:414] "containerMap: removing container" podUID="e59f243c-037f-4b74-8c60-c7fbbbc10c65" containerName="cleanup" Oct 30 17:46:18.814802 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:18.814627 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e59f243c-037f-4b74-8c60-c7fbbbc10c65" containerName="cleanup" Oct 30 17:46:18.814802 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:18.814636 4965 cpu_manager.go:414] "containerMap: removing container" podUID="74e7c579-cc3b-4bd5-998e-3bc0a50aed85" containerName="istio-proxy" Oct 30 17:46:18.814802 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:18.814642 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74e7c579-cc3b-4bd5-998e-3bc0a50aed85" containerName="istio-proxy" Oct 30 17:46:18.814802 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:18.814661 4965 cpu_manager.go:414] "containerMap: removing container" podUID="09351610-5b9d-418c-b902-0e8dcf597ab5" containerName="cleanup" Oct 30 17:46:18.814802 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:18.814665 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09351610-5b9d-418c-b902-0e8dcf597ab5" containerName="cleanup" Oct 30 17:46:18.814802 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:18.814778 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e5f291f-ff17-41bc-b87f-2beaaa451c13" containerName="cleanup" Oct 30 17:46:18.814802 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:18.814786 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="058c2414-ea11-4cd1-89e1-f4b7cb8bc0b5" containerName="cleanup" Oct 30 17:46:18.814802 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:18.814795 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fcbf5b5-34ce-408a-874b-29f6ea910d40" containerName="cleanup" Oct 30 17:46:18.814802 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:18.814802 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="1114e4ad-fc83-4876-9ffe-c47afce03fc5" containerName="istio-proxy" Oct 30 17:46:18.815116 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:18.814808 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="a628b339-91ea-46d8-9935-fc9174b9e4d2" containerName="istio-proxy" Oct 30 17:46:18.815116 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:18.814815 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="e59f243c-037f-4b74-8c60-c7fbbbc10c65" containerName="cleanup" Oct 30 17:46:18.815116 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:18.814820 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="74e7c579-cc3b-4bd5-998e-3bc0a50aed85" containerName="istio-proxy" Oct 30 17:46:18.818761 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:18.818741 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-303ba5-7bfc9db6c8-shn24" Oct 30 17:46:18.835765 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:18.835736 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-303ba5-7bfc9db6c8-shn24"] Oct 30 17:46:18.842976 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:18.842954 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-303ba5\"" Oct 30 17:46:18.843095 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:18.843035 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-303ba5-dockercfg-2mgfs\"" Oct 30 17:46:18.990896 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:18.988515 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkb8n\" (UniqueName: \"kubernetes.io/projected/0120bd60-ccb9-4b8d-99ec-04a9fba855d2-kube-api-access-jkb8n\") pod \"payload-processing-e2e-inf-b-303ba5-7bfc9db6c8-shn24\" (UID: \"0120bd60-ccb9-4b8d-99ec-04a9fba855d2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-303ba5-7bfc9db6c8-shn24" Oct 30 17:46:18.990896 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:18.988669 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0120bd60-ccb9-4b8d-99ec-04a9fba855d2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-303ba5-7bfc9db6c8-shn24\" (UID: \"0120bd60-ccb9-4b8d-99ec-04a9fba855d2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-303ba5-7bfc9db6c8-shn24" Oct 30 17:46:19.094670 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:19.091062 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkb8n\" (UniqueName: \"kubernetes.io/projected/0120bd60-ccb9-4b8d-99ec-04a9fba855d2-kube-api-access-jkb8n\") pod \"payload-processing-e2e-inf-b-303ba5-7bfc9db6c8-shn24\" (UID: \"0120bd60-ccb9-4b8d-99ec-04a9fba855d2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-303ba5-7bfc9db6c8-shn24" Oct 30 17:46:19.094670 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:19.091215 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0120bd60-ccb9-4b8d-99ec-04a9fba855d2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-303ba5-7bfc9db6c8-shn24\" (UID: \"0120bd60-ccb9-4b8d-99ec-04a9fba855d2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-303ba5-7bfc9db6c8-shn24" Oct 30 17:46:19.094670 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:19.092438 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0120bd60-ccb9-4b8d-99ec-04a9fba855d2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-303ba5-7bfc9db6c8-shn24\" (UID: \"0120bd60-ccb9-4b8d-99ec-04a9fba855d2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-303ba5-7bfc9db6c8-shn24" Oct 30 17:46:19.176883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:19.174682 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkb8n\" (UniqueName: \"kubernetes.io/projected/0120bd60-ccb9-4b8d-99ec-04a9fba855d2-kube-api-access-jkb8n\") pod \"payload-processing-e2e-inf-b-303ba5-7bfc9db6c8-shn24\" (UID: \"0120bd60-ccb9-4b8d-99ec-04a9fba855d2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-303ba5-7bfc9db6c8-shn24" Oct 30 17:46:19.433899 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:19.433783 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-303ba5-7bfc9db6c8-shn24" Oct 30 17:46:19.741543 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:46:19.741509 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0120bd60_ccb9_4b8d_99ec_04a9fba855d2.slice/crio-9856139f64755dacdf475af4fd094b31a97d8d79ebfe22a532e6e9d127235f3b WatchSource:0}: Error finding container 9856139f64755dacdf475af4fd094b31a97d8d79ebfe22a532e6e9d127235f3b: Status 404 returned error can't find the container with id 9856139f64755dacdf475af4fd094b31a97d8d79ebfe22a532e6e9d127235f3b Oct 30 17:46:19.741800 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:19.741774 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-303ba5-7bfc9db6c8-shn24"] Oct 30 17:46:20.460733 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:20.460696 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-303ba5-7bfc9db6c8-shn24" event={"ID":"0120bd60-ccb9-4b8d-99ec-04a9fba855d2","Type":"ContainerStarted","Data":"024fecdf02b80a7328b7f079b08f6ff16445fe37496baa454949d7e6ba1087c2"} Oct 30 17:46:20.461524 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:20.460761 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-303ba5-7bfc9db6c8-shn24" event={"ID":"0120bd60-ccb9-4b8d-99ec-04a9fba855d2","Type":"ContainerStarted","Data":"9856139f64755dacdf475af4fd094b31a97d8d79ebfe22a532e6e9d127235f3b"} Oct 30 17:46:20.461524 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:20.460787 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-303ba5-7bfc9db6c8-shn24" Oct 30 17:46:20.503078 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:20.503018 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-303ba5-7bfc9db6c8-shn24" podStartSLOduration=2.502990317 podStartE2EDuration="2.502990317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:46:20.502668107 +0000 UTC m=+2038.554768038" watchObservedRunningTime="2026-10-30 17:46:20.502990317 +0000 UTC m=+2038.555090227" Oct 30 17:46:31.472031 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:31.472001 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-303ba5-7bfc9db6c8-shn24" Oct 30 17:46:33.718694 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:33.718653 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd"] Oct 30 17:46:33.725515 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:33.725490 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd" Oct 30 17:46:33.744909 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:33.744668 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd"] Oct 30 17:46:33.750970 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:33.750944 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-043253-serving-cert\"" Oct 30 17:46:33.827307 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:33.827267 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d98ac6ce-ffe4-430f-849e-91daa3b32273-metrics-certs\") pod \"maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd\" (UID: \"d98ac6ce-ffe4-430f-849e-91daa3b32273\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd" Oct 30 17:46:33.827475 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:33.827319 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d98ac6ce-ffe4-430f-849e-91daa3b32273-maas-api-tls\") pod \"maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd\" (UID: \"d98ac6ce-ffe4-430f-849e-91daa3b32273\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd" Oct 30 17:46:33.827475 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:33.827453 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsvdr\" (UniqueName: \"kubernetes.io/projected/d98ac6ce-ffe4-430f-849e-91daa3b32273-kube-api-access-jsvdr\") pod \"maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd\" (UID: \"d98ac6ce-ffe4-430f-849e-91daa3b32273\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd" Oct 30 17:46:33.827551 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:33.827483 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d98ac6ce-ffe4-430f-849e-91daa3b32273-openshift-service-ca\") pod \"maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd\" (UID: \"d98ac6ce-ffe4-430f-849e-91daa3b32273\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd" Oct 30 17:46:33.962210 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:33.962170 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d98ac6ce-ffe4-430f-849e-91daa3b32273-metrics-certs\") pod \"maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd\" (UID: \"d98ac6ce-ffe4-430f-849e-91daa3b32273\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd" Oct 30 17:46:33.987053 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:33.986972 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d98ac6ce-ffe4-430f-849e-91daa3b32273-maas-api-tls\") pod \"maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd\" (UID: \"d98ac6ce-ffe4-430f-849e-91daa3b32273\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd" Oct 30 17:46:33.987943 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:33.987904 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsvdr\" (UniqueName: \"kubernetes.io/projected/d98ac6ce-ffe4-430f-849e-91daa3b32273-kube-api-access-jsvdr\") pod \"maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd\" (UID: \"d98ac6ce-ffe4-430f-849e-91daa3b32273\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd" Oct 30 17:46:33.988482 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:33.988456 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d98ac6ce-ffe4-430f-849e-91daa3b32273-openshift-service-ca\") pod \"maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd\" (UID: \"d98ac6ce-ffe4-430f-849e-91daa3b32273\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd" Oct 30 17:46:33.989724 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:33.989348 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d98ac6ce-ffe4-430f-849e-91daa3b32273-openshift-service-ca\") pod \"maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd\" (UID: \"d98ac6ce-ffe4-430f-849e-91daa3b32273\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd" Oct 30 17:46:34.021762 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:34.021723 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d98ac6ce-ffe4-430f-849e-91daa3b32273-metrics-certs\") pod \"maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd\" (UID: \"d98ac6ce-ffe4-430f-849e-91daa3b32273\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd" Oct 30 17:46:34.022805 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:34.022782 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d98ac6ce-ffe4-430f-849e-91daa3b32273-maas-api-tls\") pod \"maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd\" (UID: \"d98ac6ce-ffe4-430f-849e-91daa3b32273\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd" Oct 30 17:46:34.028607 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:34.028581 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsvdr\" (UniqueName: \"kubernetes.io/projected/d98ac6ce-ffe4-430f-849e-91daa3b32273-kube-api-access-jsvdr\") pod \"maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd\" (UID: \"d98ac6ce-ffe4-430f-849e-91daa3b32273\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd" Oct 30 17:46:34.039887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:34.039851 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd" Oct 30 17:46:34.362476 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:46:34.362434 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd98ac6ce_ffe4_430f_849e_91daa3b32273.slice/crio-ae4e2da74956329a76b4d9f91fc5ea9f9a11b6fc73c3e43e02642be4f72b6cd4 WatchSource:0}: Error finding container ae4e2da74956329a76b4d9f91fc5ea9f9a11b6fc73c3e43e02642be4f72b6cd4: Status 404 returned error can't find the container with id ae4e2da74956329a76b4d9f91fc5ea9f9a11b6fc73c3e43e02642be4f72b6cd4 Oct 30 17:46:34.362613 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:34.362521 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd"] Oct 30 17:46:34.592918 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:34.592880 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd" event={"ID":"d98ac6ce-ffe4-430f-849e-91daa3b32273","Type":"ContainerStarted","Data":"ae4e2da74956329a76b4d9f91fc5ea9f9a11b6fc73c3e43e02642be4f72b6cd4"} Oct 30 17:46:35.641410 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:35.641357 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd" event={"ID":"d98ac6ce-ffe4-430f-849e-91daa3b32273","Type":"ContainerStarted","Data":"2fc38dd7995d081ef6a155e5c28e1966fe84b04476c22294bfd0be5d7ecdbb57"} Oct 30 17:46:35.644680 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:35.644647 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd" Oct 30 17:46:35.691160 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:35.691109 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd" podStartSLOduration=2.273550059 podStartE2EDuration="2.691095633s" totalImagesPullingTime="417.545574ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:46:35.688326474 +0000 UTC m=+2053.740426405" watchObservedRunningTime="2026-10-30 17:46:35.691095633 +0000 UTC m=+2053.743195564" Oct 30 17:46:35.998376 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:46:35.998292 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:46:37.229612 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:37.229578 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:46:42.661162 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:46:42.661126 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd" Oct 30 17:47:00.361491 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:00.361454 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx"] Oct 30 17:47:00.366392 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:00.366344 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx" Oct 30 17:47:00.393488 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:00.393170 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-303ba5-serving-cert\"" Oct 30 17:47:00.401773 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:00.401743 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx"] Oct 30 17:47:00.467174 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:00.467140 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee3c3584-46f0-432e-b703-702207a1bca0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx\" (UID: \"ee3c3584-46f0-432e-b703-702207a1bca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx" Oct 30 17:47:00.467385 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:00.467345 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pb4b\" (UniqueName: \"kubernetes.io/projected/ee3c3584-46f0-432e-b703-702207a1bca0-kube-api-access-9pb4b\") pod \"maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx\" (UID: \"ee3c3584-46f0-432e-b703-702207a1bca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx" Oct 30 17:47:00.467504 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:00.467486 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee3c3584-46f0-432e-b703-702207a1bca0-metrics-certs\") pod \"maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx\" (UID: \"ee3c3584-46f0-432e-b703-702207a1bca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx" Oct 30 17:47:00.467618 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:00.467602 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee3c3584-46f0-432e-b703-702207a1bca0-maas-api-tls\") pod \"maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx\" (UID: \"ee3c3584-46f0-432e-b703-702207a1bca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx" Oct 30 17:47:00.595892 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:00.579278 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee3c3584-46f0-432e-b703-702207a1bca0-maas-api-tls\") pod \"maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx\" (UID: \"ee3c3584-46f0-432e-b703-702207a1bca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx" Oct 30 17:47:00.596339 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:00.596300 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee3c3584-46f0-432e-b703-702207a1bca0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx\" (UID: \"ee3c3584-46f0-432e-b703-702207a1bca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx" Oct 30 17:47:00.596717 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:00.596687 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pb4b\" (UniqueName: \"kubernetes.io/projected/ee3c3584-46f0-432e-b703-702207a1bca0-kube-api-access-9pb4b\") pod \"maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx\" (UID: \"ee3c3584-46f0-432e-b703-702207a1bca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx" Oct 30 17:47:00.597027 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:00.597005 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee3c3584-46f0-432e-b703-702207a1bca0-metrics-certs\") pod \"maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx\" (UID: \"ee3c3584-46f0-432e-b703-702207a1bca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx" Oct 30 17:47:00.620477 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:00.620399 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee3c3584-46f0-432e-b703-702207a1bca0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx\" (UID: \"ee3c3584-46f0-432e-b703-702207a1bca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx" Oct 30 17:47:00.671369 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:00.670537 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee3c3584-46f0-432e-b703-702207a1bca0-maas-api-tls\") pod \"maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx\" (UID: \"ee3c3584-46f0-432e-b703-702207a1bca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx" Oct 30 17:47:00.672211 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:00.672151 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee3c3584-46f0-432e-b703-702207a1bca0-metrics-certs\") pod \"maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx\" (UID: \"ee3c3584-46f0-432e-b703-702207a1bca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx" Oct 30 17:47:00.673100 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:00.672941 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pb4b\" (UniqueName: \"kubernetes.io/projected/ee3c3584-46f0-432e-b703-702207a1bca0-kube-api-access-9pb4b\") pod \"maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx\" (UID: \"ee3c3584-46f0-432e-b703-702207a1bca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx" Oct 30 17:47:00.691638 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:00.691602 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx" Oct 30 17:47:00.880285 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:00.880199 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5"] Oct 30 17:47:00.880437 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:00.880397 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" podUID="0a692092-9a09-4a93-b285-675f8268eb68" containerName="istio-proxy" containerID="cri-o://8047491db76a2f38b2c4657a8a823a1aaadd26f2ba35cbe6992ddf36d82d1e91" gracePeriod=30 Oct 30 17:47:00.894459 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:00.894429 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774"] Oct 30 17:47:00.894707 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:00.894670 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" podUID="337938ce-7eb2-45bf-a5aa-0ef36cee9444" containerName="istio-proxy" containerID="cri-o://88bb0c7d08d4ec1e0f0add33fa0bfda8420f4953b1d3ad353f0f1fa163594a97" gracePeriod=30 Oct 30 17:47:00.899269 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:00.899241 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m"] Oct 30 17:47:00.899470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:00.899431 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" podUID="d054b35e-8973-4d7b-84e5-6a49ac14bfe1" containerName="istio-proxy" containerID="cri-o://deb2369a26a05fd0b308a505763c7e9a7a6a472dde1de9fa147fea3f28af0fe9" gracePeriod=30 Oct 30 17:47:01.018443 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:47:01.018411 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee3c3584_46f0_432e_b703_702207a1bca0.slice/crio-aaf8048580fe7bd26daad1c2c1b9928aa414dbf1dd6d442723d46bf86e6f29e8 WatchSource:0}: Error finding container aaf8048580fe7bd26daad1c2c1b9928aa414dbf1dd6d442723d46bf86e6f29e8: Status 404 returned error can't find the container with id aaf8048580fe7bd26daad1c2c1b9928aa414dbf1dd6d442723d46bf86e6f29e8 Oct 30 17:47:01.019361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:01.019331 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx"] Oct 30 17:47:01.951461 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:01.951408 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx" event={"ID":"ee3c3584-46f0-432e-b703-702207a1bca0","Type":"ContainerStarted","Data":"64d8d8e72eeddfcef6fab23f72321aaa48e2b6fac464d17b0ea57584afeee88d"} Oct 30 17:47:01.951939 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:01.951533 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx" event={"ID":"ee3c3584-46f0-432e-b703-702207a1bca0","Type":"ContainerStarted","Data":"aaf8048580fe7bd26daad1c2c1b9928aa414dbf1dd6d442723d46bf86e6f29e8"} Oct 30 17:47:01.951939 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:01.951592 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx" Oct 30 17:47:01.994694 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:01.994627 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx" podStartSLOduration=1.599476833 podStartE2EDuration="1.994606754s" totalImagesPullingTime="395.129921ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:47:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:47:01.993530992 +0000 UTC m=+2080.045630922" watchObservedRunningTime="2026-10-30 17:47:01.994606754 +0000 UTC m=+2080.046706687" Oct 30 17:47:05.793274 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:05.793238 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-xmr9m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:47:05.793719 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:05.793303 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" podUID="d054b35e-8973-4d7b-84e5-6a49ac14bfe1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:47:06.072185 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:47:06.072145 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:47:06.314715 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.314685 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:47:06.318785 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.318761 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:47:06.323320 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.323298 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:47:06.360143 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.360117 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-workload-certs\") pod \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " Oct 30 17:47:06.360304 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.360153 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-credential-socket\") pod \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " Oct 30 17:47:06.360304 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.360186 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istiod-ca-cert\") pod \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " Oct 30 17:47:06.360304 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.360223 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istio-envoy\") pod \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " Oct 30 17:47:06.360304 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.360249 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-workload-certs\") pod \"0a692092-9a09-4a93-b285-675f8268eb68\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " Oct 30 17:47:06.360304 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.360270 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istio-data\") pod \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " Oct 30 17:47:06.360304 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.360296 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-istio-data\") pod \"0a692092-9a09-4a93-b285-675f8268eb68\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " Oct 30 17:47:06.360642 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.360332 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-credential-socket\") pod \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " Oct 30 17:47:06.360642 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.360351 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a692092-9a09-4a93-b285-675f8268eb68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a692092-9a09-4a93-b285-675f8268eb68-istiod-ca-cert\") pod \"0a692092-9a09-4a93-b285-675f8268eb68\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " Oct 30 17:47:06.360642 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.360378 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a692092-9a09-4a93-b285-675f8268eb68-istio-token\" (UniqueName: \"kubernetes.io/projected/0a692092-9a09-4a93-b285-675f8268eb68-istio-token\") pod \"0a692092-9a09-4a93-b285-675f8268eb68\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " Oct 30 17:47:06.360642 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.360416 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-workload-socket\") pod \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " Oct 30 17:47:06.360642 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.360415 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-workload-certs" pod "d054b35e-8973-4d7b-84e5-6a49ac14bfe1" (UID: "d054b35e-8973-4d7b-84e5-6a49ac14bfe1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:47:06.360642 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.360486 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-kube-api-access-2qlg6\" (UniqueName: \"kubernetes.io/projected/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-kube-api-access-2qlg6\") pod \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " Oct 30 17:47:06.360642 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.360514 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-credential-socket\") pod \"0a692092-9a09-4a93-b285-675f8268eb68\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " Oct 30 17:47:06.360642 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.360547 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a692092-9a09-4a93-b285-675f8268eb68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a692092-9a09-4a93-b285-675f8268eb68-istio-podinfo\") pod \"0a692092-9a09-4a93-b285-675f8268eb68\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " Oct 30 17:47:06.360642 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.360566 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a692092-9a09-4a93-b285-675f8268eb68-kube-api-access-tnq64\" (UniqueName: \"kubernetes.io/projected/0a692092-9a09-4a93-b285-675f8268eb68-kube-api-access-tnq64\") pod \"0a692092-9a09-4a93-b285-675f8268eb68\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " Oct 30 17:47:06.360642 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.360588 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istio-envoy\") pod \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " Oct 30 17:47:06.360642 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.360601 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istio-data" pod "d054b35e-8973-4d7b-84e5-6a49ac14bfe1" (UID: "d054b35e-8973-4d7b-84e5-6a49ac14bfe1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:47:06.360642 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.360643 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-workload-socket\") pod \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " Oct 30 17:47:06.361278 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.360665 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istio-data\") pod \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " Oct 30 17:47:06.361278 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.360679 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-workload-certs" pod "0a692092-9a09-4a93-b285-675f8268eb68" (UID: "0a692092-9a09-4a93-b285-675f8268eb68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:47:06.361278 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.360712 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-istio-data" pod "0a692092-9a09-4a93-b285-675f8268eb68" (UID: "0a692092-9a09-4a93-b285-675f8268eb68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:47:06.361278 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.360910 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-workload-socket" pod "337938ce-7eb2-45bf-a5aa-0ef36cee9444" (UID: "337938ce-7eb2-45bf-a5aa-0ef36cee9444"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:47:06.361278 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.360986 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istiod-ca-cert" pod "337938ce-7eb2-45bf-a5aa-0ef36cee9444" (UID: "337938ce-7eb2-45bf-a5aa-0ef36cee9444"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:47:06.361278 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.361019 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-credential-socket" pod "d054b35e-8973-4d7b-84e5-6a49ac14bfe1" (UID: "d054b35e-8973-4d7b-84e5-6a49ac14bfe1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:47:06.361278 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.361275 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-credential-socket" pod "0a692092-9a09-4a93-b285-675f8268eb68" (UID: "0a692092-9a09-4a93-b285-675f8268eb68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:47:06.361643 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.361346 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istio-data" pod "337938ce-7eb2-45bf-a5aa-0ef36cee9444" (UID: "337938ce-7eb2-45bf-a5aa-0ef36cee9444"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:47:06.361643 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.361436 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-workload-socket" pod "d054b35e-8973-4d7b-84e5-6a49ac14bfe1" (UID: "d054b35e-8973-4d7b-84e5-6a49ac14bfe1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:47:06.361643 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.360688 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-istio-envoy\") pod \"0a692092-9a09-4a93-b285-675f8268eb68\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " Oct 30 17:47:06.361643 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.361576 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-workload-socket\") pod \"0a692092-9a09-4a93-b285-675f8268eb68\" (UID: \"0a692092-9a09-4a93-b285-675f8268eb68\") " Oct 30 17:47:06.361834 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.361622 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a692092-9a09-4a93-b285-675f8268eb68-istiod-ca-cert" pod "0a692092-9a09-4a93-b285-675f8268eb68" (UID: "0a692092-9a09-4a93-b285-675f8268eb68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:47:06.361834 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.361694 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istio-podinfo\") pod \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " Oct 30 17:47:06.361834 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.361776 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-workload-certs\") pod \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " Oct 30 17:47:06.361834 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.361798 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-credential-socket" pod "337938ce-7eb2-45bf-a5aa-0ef36cee9444" (UID: "337938ce-7eb2-45bf-a5aa-0ef36cee9444"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:47:06.362091 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.361836 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istio-token\" (UniqueName: \"kubernetes.io/projected/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istio-token\") pod \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " Oct 30 17:47:06.362091 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.361880 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istio-token\" (UniqueName: \"kubernetes.io/projected/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istio-token\") pod \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " Oct 30 17:47:06.362091 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.361959 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/337938ce-7eb2-45bf-a5aa-0ef36cee9444-kube-api-access-7dg4j\" (UniqueName: \"kubernetes.io/projected/337938ce-7eb2-45bf-a5aa-0ef36cee9444-kube-api-access-7dg4j\") pod \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\" (UID: \"337938ce-7eb2-45bf-a5aa-0ef36cee9444\") " Oct 30 17:47:06.362091 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.362003 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istiod-ca-cert\") pod \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " Oct 30 17:47:06.362091 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.362029 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istio-podinfo\") pod \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\" (UID: \"d054b35e-8973-4d7b-84e5-6a49ac14bfe1\") " Oct 30 17:47:06.362091 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.362072 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-workload-certs" pod "337938ce-7eb2-45bf-a5aa-0ef36cee9444" (UID: "337938ce-7eb2-45bf-a5aa-0ef36cee9444"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:47:06.362407 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.362330 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-workload-socket" pod "0a692092-9a09-4a93-b285-675f8268eb68" (UID: "0a692092-9a09-4a93-b285-675f8268eb68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:47:06.363325 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.363299 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istiod-ca-cert" pod "d054b35e-8973-4d7b-84e5-6a49ac14bfe1" (UID: "d054b35e-8973-4d7b-84e5-6a49ac14bfe1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:47:06.363586 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.363566 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:06.364062 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.363593 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:06.364062 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.363623 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:06.364062 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.363638 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:06.364062 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.363655 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a692092-9a09-4a93-b285-675f8268eb68-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:06.364062 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.363668 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:06.364062 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.363680 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:06.364062 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.363692 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:06.364062 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.363705 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:06.364062 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.363717 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:06.364062 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.363730 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:06.364062 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.363745 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:06.364062 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.363759 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:06.364062 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.363772 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:06.364062 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.363784 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:06.364680 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.364228 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a692092-9a09-4a93-b285-675f8268eb68-istio-token" pod "0a692092-9a09-4a93-b285-675f8268eb68" (UID: "0a692092-9a09-4a93-b285-675f8268eb68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:47:06.364680 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.364309 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-kube-api-access-2qlg6" pod "d054b35e-8973-4d7b-84e5-6a49ac14bfe1" (UID: "d054b35e-8973-4d7b-84e5-6a49ac14bfe1"). InnerVolumeSpecName "kube-api-access-2qlg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:47:06.364928 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.364901 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istio-podinfo" pod "d054b35e-8973-4d7b-84e5-6a49ac14bfe1" (UID: "d054b35e-8973-4d7b-84e5-6a49ac14bfe1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:47:06.365969 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.365937 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a692092-9a09-4a93-b285-675f8268eb68-kube-api-access-tnq64" pod "0a692092-9a09-4a93-b285-675f8268eb68" (UID: "0a692092-9a09-4a93-b285-675f8268eb68"). InnerVolumeSpecName "kube-api-access-tnq64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:47:06.366087 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.366032 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a692092-9a09-4a93-b285-675f8268eb68-istio-podinfo" pod "0a692092-9a09-4a93-b285-675f8268eb68" (UID: "0a692092-9a09-4a93-b285-675f8268eb68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:47:06.366414 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.366364 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istio-podinfo" pod "337938ce-7eb2-45bf-a5aa-0ef36cee9444" (UID: "337938ce-7eb2-45bf-a5aa-0ef36cee9444"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:47:06.366530 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.366376 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istio-token" pod "d054b35e-8973-4d7b-84e5-6a49ac14bfe1" (UID: "d054b35e-8973-4d7b-84e5-6a49ac14bfe1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:47:06.366642 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.366620 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istio-envoy" pod "337938ce-7eb2-45bf-a5aa-0ef36cee9444" (UID: "337938ce-7eb2-45bf-a5aa-0ef36cee9444"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:47:06.366715 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.366641 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istio-envoy" pod "d054b35e-8973-4d7b-84e5-6a49ac14bfe1" (UID: "d054b35e-8973-4d7b-84e5-6a49ac14bfe1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:47:06.366889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.366831 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istio-token" pod "337938ce-7eb2-45bf-a5aa-0ef36cee9444" (UID: "337938ce-7eb2-45bf-a5aa-0ef36cee9444"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:47:06.367022 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.367000 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-istio-envoy" pod "0a692092-9a09-4a93-b285-675f8268eb68" (UID: "0a692092-9a09-4a93-b285-675f8268eb68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:47:06.367457 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.367441 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/337938ce-7eb2-45bf-a5aa-0ef36cee9444-kube-api-access-7dg4j" pod "337938ce-7eb2-45bf-a5aa-0ef36cee9444" (UID: "337938ce-7eb2-45bf-a5aa-0ef36cee9444"). InnerVolumeSpecName "kube-api-access-7dg4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:47:06.466098 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.466063 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnq64\" (UniqueName: \"kubernetes.io/projected/0a692092-9a09-4a93-b285-675f8268eb68-kube-api-access-tnq64\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:06.466098 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.466098 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:06.466362 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.466113 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a692092-9a09-4a93-b285-675f8268eb68-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:06.466362 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.466126 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:06.466362 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.466138 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/337938ce-7eb2-45bf-a5aa-0ef36cee9444-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:06.466362 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.466150 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:06.466362 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.466160 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dg4j\" (UniqueName: \"kubernetes.io/projected/337938ce-7eb2-45bf-a5aa-0ef36cee9444-kube-api-access-7dg4j\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:06.466362 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.466169 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:06.466362 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.466182 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:06.466362 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.466194 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a692092-9a09-4a93-b285-675f8268eb68-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:06.466362 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.466209 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qlg6\" (UniqueName: \"kubernetes.io/projected/d054b35e-8973-4d7b-84e5-6a49ac14bfe1-kube-api-access-2qlg6\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:06.466362 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:06.466219 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a692092-9a09-4a93-b285-675f8268eb68-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:07.042725 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:07.042690 4965 generic.go:356] "Generic (PLEG): container finished" podID="d054b35e-8973-4d7b-84e5-6a49ac14bfe1" containerID="deb2369a26a05fd0b308a505763c7e9a7a6a472dde1de9fa147fea3f28af0fe9" exitCode=0 Oct 30 17:47:07.043240 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:07.042738 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" Oct 30 17:47:07.043240 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:07.042772 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" event={"ID":"d054b35e-8973-4d7b-84e5-6a49ac14bfe1","Type":"ContainerDied","Data":"deb2369a26a05fd0b308a505763c7e9a7a6a472dde1de9fa147fea3f28af0fe9"} Oct 30 17:47:07.043240 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:07.042828 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m" event={"ID":"d054b35e-8973-4d7b-84e5-6a49ac14bfe1","Type":"ContainerDied","Data":"9c455729f4480efc778983c52e6ded4dc15a4b0388209ef53c10e047cda5188f"} Oct 30 17:47:07.043240 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:07.042852 4965 scope.go:122] "RemoveContainer" containerID="deb2369a26a05fd0b308a505763c7e9a7a6a472dde1de9fa147fea3f28af0fe9" Oct 30 17:47:07.044456 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:07.044434 4965 generic.go:356] "Generic (PLEG): container finished" podID="0a692092-9a09-4a93-b285-675f8268eb68" containerID="8047491db76a2f38b2c4657a8a823a1aaadd26f2ba35cbe6992ddf36d82d1e91" exitCode=0 Oct 30 17:47:07.044574 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:07.044478 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" Oct 30 17:47:07.044574 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:07.044524 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" event={"ID":"0a692092-9a09-4a93-b285-675f8268eb68","Type":"ContainerDied","Data":"8047491db76a2f38b2c4657a8a823a1aaadd26f2ba35cbe6992ddf36d82d1e91"} Oct 30 17:47:07.044686 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:07.044580 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5" event={"ID":"0a692092-9a09-4a93-b285-675f8268eb68","Type":"ContainerDied","Data":"39e8282a6e83d06085e0f8612cd129e1fe467156fd08e1569e5e8436397af189"} Oct 30 17:47:07.046203 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:07.046184 4965 generic.go:356] "Generic (PLEG): container finished" podID="337938ce-7eb2-45bf-a5aa-0ef36cee9444" containerID="88bb0c7d08d4ec1e0f0add33fa0bfda8420f4953b1d3ad353f0f1fa163594a97" exitCode=0 Oct 30 17:47:07.046330 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:07.046239 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" Oct 30 17:47:07.046330 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:07.046290 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" event={"ID":"337938ce-7eb2-45bf-a5aa-0ef36cee9444","Type":"ContainerDied","Data":"88bb0c7d08d4ec1e0f0add33fa0bfda8420f4953b1d3ad353f0f1fa163594a97"} Oct 30 17:47:07.046421 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:07.046334 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774" event={"ID":"337938ce-7eb2-45bf-a5aa-0ef36cee9444","Type":"ContainerDied","Data":"ec369bb42765200f9654e0bd2cc0db0e3dcdf0c8d4b94dc042f37f8a2f60faf4"} Oct 30 17:47:07.059256 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:07.058848 4965 scope.go:122] "RemoveContainer" containerID="deb2369a26a05fd0b308a505763c7e9a7a6a472dde1de9fa147fea3f28af0fe9" Oct 30 17:47:07.059256 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:47:07.059204 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deb2369a26a05fd0b308a505763c7e9a7a6a472dde1de9fa147fea3f28af0fe9\": container with ID starting with deb2369a26a05fd0b308a505763c7e9a7a6a472dde1de9fa147fea3f28af0fe9 not found: ID does not exist" containerID="deb2369a26a05fd0b308a505763c7e9a7a6a472dde1de9fa147fea3f28af0fe9" Oct 30 17:47:07.059256 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:07.059229 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deb2369a26a05fd0b308a505763c7e9a7a6a472dde1de9fa147fea3f28af0fe9"} err="failed to get container status \"deb2369a26a05fd0b308a505763c7e9a7a6a472dde1de9fa147fea3f28af0fe9\": rpc error: code = NotFound desc = could not find container \"deb2369a26a05fd0b308a505763c7e9a7a6a472dde1de9fa147fea3f28af0fe9\": container with ID starting with deb2369a26a05fd0b308a505763c7e9a7a6a472dde1de9fa147fea3f28af0fe9 not found: ID does not exist" Oct 30 17:47:07.059256 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:07.059246 4965 scope.go:122] "RemoveContainer" containerID="8047491db76a2f38b2c4657a8a823a1aaadd26f2ba35cbe6992ddf36d82d1e91" Oct 30 17:47:07.070633 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:07.070618 4965 scope.go:122] "RemoveContainer" containerID="8047491db76a2f38b2c4657a8a823a1aaadd26f2ba35cbe6992ddf36d82d1e91" Oct 30 17:47:07.070899 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:47:07.070853 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8047491db76a2f38b2c4657a8a823a1aaadd26f2ba35cbe6992ddf36d82d1e91\": container with ID starting with 8047491db76a2f38b2c4657a8a823a1aaadd26f2ba35cbe6992ddf36d82d1e91 not found: ID does not exist" containerID="8047491db76a2f38b2c4657a8a823a1aaadd26f2ba35cbe6992ddf36d82d1e91" Oct 30 17:47:07.070946 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:07.070906 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8047491db76a2f38b2c4657a8a823a1aaadd26f2ba35cbe6992ddf36d82d1e91"} err="failed to get container status \"8047491db76a2f38b2c4657a8a823a1aaadd26f2ba35cbe6992ddf36d82d1e91\": rpc error: code = NotFound desc = could not find container \"8047491db76a2f38b2c4657a8a823a1aaadd26f2ba35cbe6992ddf36d82d1e91\": container with ID starting with 8047491db76a2f38b2c4657a8a823a1aaadd26f2ba35cbe6992ddf36d82d1e91 not found: ID does not exist" Oct 30 17:47:07.070946 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:07.070920 4965 scope.go:122] "RemoveContainer" containerID="88bb0c7d08d4ec1e0f0add33fa0bfda8420f4953b1d3ad353f0f1fa163594a97" Oct 30 17:47:07.080976 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:07.080957 4965 scope.go:122] "RemoveContainer" containerID="88bb0c7d08d4ec1e0f0add33fa0bfda8420f4953b1d3ad353f0f1fa163594a97" Oct 30 17:47:07.081243 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:47:07.081225 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88bb0c7d08d4ec1e0f0add33fa0bfda8420f4953b1d3ad353f0f1fa163594a97\": container with ID starting with 88bb0c7d08d4ec1e0f0add33fa0bfda8420f4953b1d3ad353f0f1fa163594a97 not found: ID does not exist" containerID="88bb0c7d08d4ec1e0f0add33fa0bfda8420f4953b1d3ad353f0f1fa163594a97" Oct 30 17:47:07.081291 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:07.081249 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88bb0c7d08d4ec1e0f0add33fa0bfda8420f4953b1d3ad353f0f1fa163594a97"} err="failed to get container status \"88bb0c7d08d4ec1e0f0add33fa0bfda8420f4953b1d3ad353f0f1fa163594a97\": rpc error: code = NotFound desc = could not find container \"88bb0c7d08d4ec1e0f0add33fa0bfda8420f4953b1d3ad353f0f1fa163594a97\": container with ID starting with 88bb0c7d08d4ec1e0f0add33fa0bfda8420f4953b1d3ad353f0f1fa163594a97 not found: ID does not exist" Oct 30 17:47:07.099733 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:07.099710 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5"] Oct 30 17:47:07.111011 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:07.110990 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-zv8m5"] Oct 30 17:47:07.138404 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:07.138381 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m"] Oct 30 17:47:07.141993 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:07.141974 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xmr9m"] Oct 30 17:47:07.190526 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:07.190420 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774"] Oct 30 17:47:07.194345 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:07.194321 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rh774"] Oct 30 17:47:07.966490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:07.966462 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx" Oct 30 17:47:08.103738 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:08.103706 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a692092-9a09-4a93-b285-675f8268eb68" path="/var/lib/kubelet/pods/0a692092-9a09-4a93-b285-675f8268eb68/volumes" Oct 30 17:47:08.104458 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:08.104435 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="337938ce-7eb2-45bf-a5aa-0ef36cee9444" path="/var/lib/kubelet/pods/337938ce-7eb2-45bf-a5aa-0ef36cee9444/volumes" Oct 30 17:47:08.104979 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:08.104959 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d054b35e-8973-4d7b-84e5-6a49ac14bfe1" path="/var/lib/kubelet/pods/d054b35e-8973-4d7b-84e5-6a49ac14bfe1/volumes" Oct 30 17:47:14.278656 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:14.278621 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b"] Oct 30 17:47:22.443144 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:22.443109 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 17:47:22.445260 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:22.445228 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 17:47:29.336299 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:29.336161 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b" podUID="4f458064-ff24-4cf6-acb5-75830949042a" containerName="main" containerID="cri-o://a3753c57584061b932dd703207ea84fc3e01ac35fb88ad3dc96eaed4aae24293" gracePeriod=45 Oct 30 17:47:29.661241 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:29.661213 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b" Oct 30 17:47:29.842937 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:29.842893 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f458064-ff24-4cf6-acb5-75830949042a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f458064-ff24-4cf6-acb5-75830949042a-model-cache\") pod \"4f458064-ff24-4cf6-acb5-75830949042a\" (UID: \"4f458064-ff24-4cf6-acb5-75830949042a\") " Oct 30 17:47:29.843124 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:29.842956 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f458064-ff24-4cf6-acb5-75830949042a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f458064-ff24-4cf6-acb5-75830949042a-dshm\") pod \"4f458064-ff24-4cf6-acb5-75830949042a\" (UID: \"4f458064-ff24-4cf6-acb5-75830949042a\") " Oct 30 17:47:29.843124 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:29.843009 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f458064-ff24-4cf6-acb5-75830949042a-home\" (UniqueName: \"kubernetes.io/empty-dir/4f458064-ff24-4cf6-acb5-75830949042a-home\") pod \"4f458064-ff24-4cf6-acb5-75830949042a\" (UID: \"4f458064-ff24-4cf6-acb5-75830949042a\") " Oct 30 17:47:29.843124 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:29.843035 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f458064-ff24-4cf6-acb5-75830949042a-kube-api-access-lcv4j\" (UniqueName: \"kubernetes.io/projected/4f458064-ff24-4cf6-acb5-75830949042a-kube-api-access-lcv4j\") pod \"4f458064-ff24-4cf6-acb5-75830949042a\" (UID: \"4f458064-ff24-4cf6-acb5-75830949042a\") " Oct 30 17:47:29.843124 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:29.843062 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f458064-ff24-4cf6-acb5-75830949042a-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f458064-ff24-4cf6-acb5-75830949042a-tls-certs\") pod \"4f458064-ff24-4cf6-acb5-75830949042a\" (UID: \"4f458064-ff24-4cf6-acb5-75830949042a\") " Oct 30 17:47:29.843359 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:29.843157 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f458064-ff24-4cf6-acb5-75830949042a-model-cache" pod "4f458064-ff24-4cf6-acb5-75830949042a" (UID: "4f458064-ff24-4cf6-acb5-75830949042a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:47:29.843359 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:29.843282 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f458064-ff24-4cf6-acb5-75830949042a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f458064-ff24-4cf6-acb5-75830949042a-tmp-dir\") pod \"4f458064-ff24-4cf6-acb5-75830949042a\" (UID: \"4f458064-ff24-4cf6-acb5-75830949042a\") " Oct 30 17:47:29.843359 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:29.843335 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f458064-ff24-4cf6-acb5-75830949042a-home" pod "4f458064-ff24-4cf6-acb5-75830949042a" (UID: "4f458064-ff24-4cf6-acb5-75830949042a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:47:29.843536 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:29.843485 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f458064-ff24-4cf6-acb5-75830949042a-tmp-dir" pod "4f458064-ff24-4cf6-acb5-75830949042a" (UID: "4f458064-ff24-4cf6-acb5-75830949042a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:47:29.843951 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:29.843930 4965 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f458064-ff24-4cf6-acb5-75830949042a-model-cache\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:29.843951 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:29.843949 4965 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f458064-ff24-4cf6-acb5-75830949042a-home\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:29.844132 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:29.843966 4965 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f458064-ff24-4cf6-acb5-75830949042a-tmp-dir\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:29.845422 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:29.845391 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f458064-ff24-4cf6-acb5-75830949042a-kube-api-access-lcv4j" pod "4f458064-ff24-4cf6-acb5-75830949042a" (UID: "4f458064-ff24-4cf6-acb5-75830949042a"). InnerVolumeSpecName "kube-api-access-lcv4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:47:29.845522 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:29.845432 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f458064-ff24-4cf6-acb5-75830949042a-dshm" pod "4f458064-ff24-4cf6-acb5-75830949042a" (UID: "4f458064-ff24-4cf6-acb5-75830949042a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:47:29.845522 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:29.845479 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f458064-ff24-4cf6-acb5-75830949042a-tls-certs" pod "4f458064-ff24-4cf6-acb5-75830949042a" (UID: "4f458064-ff24-4cf6-acb5-75830949042a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:47:29.945682 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:29.945606 4965 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f458064-ff24-4cf6-acb5-75830949042a-dshm\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:29.945682 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:29.945636 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcv4j\" (UniqueName: \"kubernetes.io/projected/4f458064-ff24-4cf6-acb5-75830949042a-kube-api-access-lcv4j\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:29.945682 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:29.945648 4965 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f458064-ff24-4cf6-acb5-75830949042a-tls-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:30.301931 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:30.301838 4965 generic.go:356] "Generic (PLEG): container finished" podID="4f458064-ff24-4cf6-acb5-75830949042a" containerID="a3753c57584061b932dd703207ea84fc3e01ac35fb88ad3dc96eaed4aae24293" exitCode=0 Oct 30 17:47:30.301931 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:30.301918 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b" Oct 30 17:47:30.302125 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:30.301927 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b" event={"ID":"4f458064-ff24-4cf6-acb5-75830949042a","Type":"ContainerDied","Data":"a3753c57584061b932dd703207ea84fc3e01ac35fb88ad3dc96eaed4aae24293"} Oct 30 17:47:30.302125 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:30.301970 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b" event={"ID":"4f458064-ff24-4cf6-acb5-75830949042a","Type":"ContainerDied","Data":"f3c9881ff8ed6abef36ec1332c74a7f9750e04fc04a3e9c7a5f3344db5ffe0e5"} Oct 30 17:47:30.302125 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:30.301989 4965 scope.go:122] "RemoveContainer" containerID="a3753c57584061b932dd703207ea84fc3e01ac35fb88ad3dc96eaed4aae24293" Oct 30 17:47:30.317349 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:30.317326 4965 scope.go:122] "RemoveContainer" containerID="a3753c57584061b932dd703207ea84fc3e01ac35fb88ad3dc96eaed4aae24293" Oct 30 17:47:30.317649 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:47:30.317628 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3753c57584061b932dd703207ea84fc3e01ac35fb88ad3dc96eaed4aae24293\": container with ID starting with a3753c57584061b932dd703207ea84fc3e01ac35fb88ad3dc96eaed4aae24293 not found: ID does not exist" containerID="a3753c57584061b932dd703207ea84fc3e01ac35fb88ad3dc96eaed4aae24293" Oct 30 17:47:30.317710 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:30.317659 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3753c57584061b932dd703207ea84fc3e01ac35fb88ad3dc96eaed4aae24293"} err="failed to get container status \"a3753c57584061b932dd703207ea84fc3e01ac35fb88ad3dc96eaed4aae24293\": rpc error: code = NotFound desc = could not find container \"a3753c57584061b932dd703207ea84fc3e01ac35fb88ad3dc96eaed4aae24293\": container with ID starting with a3753c57584061b932dd703207ea84fc3e01ac35fb88ad3dc96eaed4aae24293 not found: ID does not exist" Oct 30 17:47:30.356327 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:30.356294 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b"] Oct 30 17:47:30.368723 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:30.368691 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-bba90b/test-model-bba90b-kserve-74557d96cf-cfc9b"] Oct 30 17:47:32.107298 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:32.105214 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f458064-ff24-4cf6-acb5-75830949042a" path="/var/lib/kubelet/pods/4f458064-ff24-4cf6-acb5-75830949042a/volumes" Oct 30 17:47:33.352937 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.352712 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz"] Oct 30 17:47:33.354699 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.354672 4965 cpu_manager.go:414] "containerMap: removing container" podUID="4f458064-ff24-4cf6-acb5-75830949042a" containerName="main" Oct 30 17:47:33.354856 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.354841 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f458064-ff24-4cf6-acb5-75830949042a" containerName="main" Oct 30 17:47:33.355028 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.355004 4965 cpu_manager.go:414] "containerMap: removing container" podUID="d054b35e-8973-4d7b-84e5-6a49ac14bfe1" containerName="istio-proxy" Oct 30 17:47:33.355028 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.355024 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d054b35e-8973-4d7b-84e5-6a49ac14bfe1" containerName="istio-proxy" Oct 30 17:47:33.355195 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.355058 4965 cpu_manager.go:414] "containerMap: removing container" podUID="0a692092-9a09-4a93-b285-675f8268eb68" containerName="istio-proxy" Oct 30 17:47:33.355195 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.355066 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a692092-9a09-4a93-b285-675f8268eb68" containerName="istio-proxy" Oct 30 17:47:33.355195 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.355084 4965 cpu_manager.go:414] "containerMap: removing container" podUID="337938ce-7eb2-45bf-a5aa-0ef36cee9444" containerName="istio-proxy" Oct 30 17:47:33.355195 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.355091 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="337938ce-7eb2-45bf-a5aa-0ef36cee9444" containerName="istio-proxy" Oct 30 17:47:33.355832 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.355801 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f458064-ff24-4cf6-acb5-75830949042a" containerName="main" Oct 30 17:47:33.355940 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.355836 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="d054b35e-8973-4d7b-84e5-6a49ac14bfe1" containerName="istio-proxy" Oct 30 17:47:33.355940 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.355854 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a692092-9a09-4a93-b285-675f8268eb68" containerName="istio-proxy" Oct 30 17:47:33.355940 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.355880 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="337938ce-7eb2-45bf-a5aa-0ef36cee9444" containerName="istio-proxy" Oct 30 17:47:33.361995 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.361975 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:33.380003 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.379970 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz"] Oct 30 17:47:33.408430 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.408401 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-bcdac0-openshift-default-dockercfg-6hq5t\"" Oct 30 17:47:33.499550 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.499511 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50d223e1-883f-454f-9c29-348129b7d676-istiod-ca-cert\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:33.499731 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.499668 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-workload-certs\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:33.499801 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.499738 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50d223e1-883f-454f-9c29-348129b7d676-istio-token\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:33.499801 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.499772 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-credential-socket\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:33.499901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.499804 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-istio-envoy\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:33.499901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.499838 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4np2d\" (UniqueName: \"kubernetes.io/projected/50d223e1-883f-454f-9c29-348129b7d676-kube-api-access-4np2d\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:33.499974 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.499955 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-workload-socket\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:33.500023 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.500007 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-istio-data\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:33.500128 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.500114 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50d223e1-883f-454f-9c29-348129b7d676-istio-podinfo\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:33.610449 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.608838 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-istio-data\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:33.610449 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.608983 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50d223e1-883f-454f-9c29-348129b7d676-istio-podinfo\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:33.610449 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.609149 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50d223e1-883f-454f-9c29-348129b7d676-istiod-ca-cert\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:33.610449 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.609255 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-workload-certs\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:33.610449 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.609313 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50d223e1-883f-454f-9c29-348129b7d676-istio-token\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:33.610449 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.609361 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-istio-data\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:33.610449 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.609364 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-credential-socket\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:33.610449 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.609439 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-istio-envoy\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:33.610449 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.609502 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4np2d\" (UniqueName: \"kubernetes.io/projected/50d223e1-883f-454f-9c29-348129b7d676-kube-api-access-4np2d\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:33.610449 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.609568 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-workload-socket\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:33.610449 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.609638 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-credential-socket\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:33.611230 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.610534 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-workload-socket\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:33.611230 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.611186 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50d223e1-883f-454f-9c29-348129b7d676-istiod-ca-cert\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:33.611451 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.611431 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-workload-certs\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:33.666608 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.666578 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50d223e1-883f-454f-9c29-348129b7d676-istio-podinfo\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:33.672946 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.669152 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-istio-envoy\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:33.673673 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.673643 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50d223e1-883f-454f-9c29-348129b7d676-istio-token\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:33.676706 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.676522 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4np2d\" (UniqueName: \"kubernetes.io/projected/50d223e1-883f-454f-9c29-348129b7d676-kube-api-access-4np2d\") pod \"e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:33.694556 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:33.694183 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:34.120004 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:34.119961 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz"] Oct 30 17:47:34.120122 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:47:34.120029 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50d223e1_883f_454f_9c29_348129b7d676.slice/crio-2ed6a417688766ef132e949a5bb0e61458a32d12b1f302b9948a02fd70c29c66 WatchSource:0}: Error finding container 2ed6a417688766ef132e949a5bb0e61458a32d12b1f302b9948a02fd70c29c66: Status 404 returned error can't find the container with id 2ed6a417688766ef132e949a5bb0e61458a32d12b1f302b9948a02fd70c29c66 Oct 30 17:47:34.122728 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:34.122689 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:47:34.122815 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:34.122760 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:47:34.122815 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:34.122788 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:47:34.335147 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:34.335112 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" event={"ID":"50d223e1-883f-454f-9c29-348129b7d676","Type":"ContainerStarted","Data":"0d0c0f6ce3e971f2cab20611549e4304e190a972dcf33697fd4ecc0fc7b165df"} Oct 30 17:47:34.335341 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:34.335171 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" event={"ID":"50d223e1-883f-454f-9c29-348129b7d676","Type":"ContainerStarted","Data":"2ed6a417688766ef132e949a5bb0e61458a32d12b1f302b9948a02fd70c29c66"} Oct 30 17:47:34.405936 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:34.399940 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" podStartSLOduration=1.399918341 podStartE2EDuration="1.399918341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:47:34.394238047 +0000 UTC m=+2112.446338021" watchObservedRunningTime="2026-10-30 17:47:34.399918341 +0000 UTC m=+2112.452018274" Oct 30 17:47:34.695007 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:34.694932 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:35.701916 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:35.701854 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:35.998559 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:47:35.998470 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:47:36.351802 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.351773 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:36.352977 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.352953 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:47:36.656576 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.656502 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx"] Oct 30 17:47:36.661586 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.661565 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:36.677223 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.677174 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-043253-openshift-default-dockercfg-vlpqj\"" Oct 30 17:47:36.695128 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.694986 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx"] Oct 30 17:47:36.779738 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.779700 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xn6n\" (UniqueName: \"kubernetes.io/projected/f98b7bc7-2701-4a7e-9847-131fb50d65c9-kube-api-access-9xn6n\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bjjmx\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:36.779738 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.779740 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istiod-ca-cert\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bjjmx\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:36.780215 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.779851 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istio-token\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bjjmx\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:36.780215 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.779921 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istio-envoy\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bjjmx\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:36.780215 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.779975 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istio-data\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bjjmx\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:36.780215 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.780046 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-credential-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bjjmx\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:36.780215 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.780139 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-workload-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bjjmx\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:36.780215 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.780184 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-workload-certs\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bjjmx\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:36.780431 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.780320 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istio-podinfo\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bjjmx\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:36.881709 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.881677 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istio-podinfo\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bjjmx\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:36.881912 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.881727 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xn6n\" (UniqueName: \"kubernetes.io/projected/f98b7bc7-2701-4a7e-9847-131fb50d65c9-kube-api-access-9xn6n\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bjjmx\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:36.881912 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.881745 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istiod-ca-cert\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bjjmx\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:36.881912 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.881785 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istio-token\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bjjmx\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:36.881912 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.881820 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istio-envoy\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bjjmx\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:36.881912 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.881855 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istio-data\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bjjmx\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:36.882226 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.881962 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-credential-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bjjmx\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:36.882226 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.882052 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-workload-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bjjmx\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:36.882226 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.882117 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-workload-certs\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bjjmx\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:36.882404 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.882309 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istio-data\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bjjmx\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:36.882466 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.882440 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-credential-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bjjmx\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:36.882510 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.882470 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-workload-certs\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bjjmx\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:36.882549 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.882532 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-workload-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bjjmx\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:36.882634 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.882614 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istiod-ca-cert\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bjjmx\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:36.884281 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.884253 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istio-envoy\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bjjmx\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:36.884502 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.884483 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istio-podinfo\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bjjmx\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:36.904783 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.904750 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istio-token\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bjjmx\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:36.904954 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.904935 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xn6n\" (UniqueName: \"kubernetes.io/projected/f98b7bc7-2701-4a7e-9847-131fb50d65c9-kube-api-access-9xn6n\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-bjjmx\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:36.978174 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:36.978093 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:37.309215 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:37.309193 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx"] Oct 30 17:47:37.310273 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:47:37.310242 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf98b7bc7_2701_4a7e_9847_131fb50d65c9.slice/crio-2cb8cc76fba2af46277ce7b1c5ddf63a951b644fd8882b784bc3bd77b70ce668 WatchSource:0}: Error finding container 2cb8cc76fba2af46277ce7b1c5ddf63a951b644fd8882b784bc3bd77b70ce668: Status 404 returned error can't find the container with id 2cb8cc76fba2af46277ce7b1c5ddf63a951b644fd8882b784bc3bd77b70ce668 Oct 30 17:47:37.313210 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:37.313173 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:47:37.313309 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:37.313244 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:47:37.313309 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:37.313274 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:47:37.362579 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:37.362551 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" event={"ID":"f98b7bc7-2701-4a7e-9847-131fb50d65c9","Type":"ContainerStarted","Data":"2cb8cc76fba2af46277ce7b1c5ddf63a951b644fd8882b784bc3bd77b70ce668"} Oct 30 17:47:38.172316 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:38.172280 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v"] Oct 30 17:47:38.394247 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:38.394206 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" event={"ID":"f98b7bc7-2701-4a7e-9847-131fb50d65c9","Type":"ContainerStarted","Data":"dd484f82b92d6ef0691510f7730c95b4bee030409a73e9bad066c8cfa999ca48"} Oct 30 17:47:38.451930 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:38.451813 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" podStartSLOduration=2.451799953 podStartE2EDuration="2.451799953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:47:38.448617752 +0000 UTC m=+2116.500717693" watchObservedRunningTime="2026-10-30 17:47:38.451799953 +0000 UTC m=+2116.503899883" Oct 30 17:47:38.981890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:38.981245 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:38.983533 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:38.983493 4965 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 17:47:38.983670 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:38.983558 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" podUID="f98b7bc7-2701-4a7e-9847-131fb50d65c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 17:47:39.981887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:39.981454 4965 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 17:47:39.981887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:39.981516 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" podUID="f98b7bc7-2701-4a7e-9847-131fb50d65c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 17:47:40.446113 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:40.446078 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bcdac0-698d7bc59-zr7ws"] Oct 30 17:47:40.452006 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:40.451983 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcdac0-698d7bc59-zr7ws" Oct 30 17:47:40.528523 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:40.528490 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-bcdac0-dockercfg-hh44s\"" Oct 30 17:47:40.528846 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:40.528827 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-bcdac0\"" Oct 30 17:47:40.614920 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:40.588843 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bcdac0-698d7bc59-zr7ws"] Oct 30 17:47:40.617101 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:40.617064 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af50d6f-9968-40fd-9159-b6f7078023db-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bcdac0-698d7bc59-zr7ws\" (UID: \"3af50d6f-9968-40fd-9159-b6f7078023db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcdac0-698d7bc59-zr7ws" Oct 30 17:47:40.617420 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:40.617396 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj476\" (UniqueName: \"kubernetes.io/projected/3af50d6f-9968-40fd-9159-b6f7078023db-kube-api-access-fj476\") pod \"payload-pre-processing-e2e-inf-b-bcdac0-698d7bc59-zr7ws\" (UID: \"3af50d6f-9968-40fd-9159-b6f7078023db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcdac0-698d7bc59-zr7ws" Oct 30 17:47:40.794763 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:40.794669 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af50d6f-9968-40fd-9159-b6f7078023db-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bcdac0-698d7bc59-zr7ws\" (UID: \"3af50d6f-9968-40fd-9159-b6f7078023db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcdac0-698d7bc59-zr7ws" Oct 30 17:47:40.810693 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:40.810660 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj476\" (UniqueName: \"kubernetes.io/projected/3af50d6f-9968-40fd-9159-b6f7078023db-kube-api-access-fj476\") pod \"payload-pre-processing-e2e-inf-b-bcdac0-698d7bc59-zr7ws\" (UID: \"3af50d6f-9968-40fd-9159-b6f7078023db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcdac0-698d7bc59-zr7ws" Oct 30 17:47:40.811617 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:40.807613 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af50d6f-9968-40fd-9159-b6f7078023db-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bcdac0-698d7bc59-zr7ws\" (UID: \"3af50d6f-9968-40fd-9159-b6f7078023db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcdac0-698d7bc59-zr7ws" Oct 30 17:47:40.950961 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:40.950923 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj476\" (UniqueName: \"kubernetes.io/projected/3af50d6f-9968-40fd-9159-b6f7078023db-kube-api-access-fj476\") pod \"payload-pre-processing-e2e-inf-b-bcdac0-698d7bc59-zr7ws\" (UID: \"3af50d6f-9968-40fd-9159-b6f7078023db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcdac0-698d7bc59-zr7ws" Oct 30 17:47:40.982825 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:40.982485 4965 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 17:47:40.982825 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:40.982540 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" podUID="f98b7bc7-2701-4a7e-9847-131fb50d65c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 17:47:41.216611 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:41.216575 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcdac0-698d7bc59-zr7ws" Oct 30 17:47:41.577885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:41.559079 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bcdac0-698d7bc59-zr7ws"] Oct 30 17:47:41.981897 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:41.981212 4965 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 17:47:41.981897 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:41.981269 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" podUID="f98b7bc7-2701-4a7e-9847-131fb50d65c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 17:47:42.507945 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:42.507857 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcdac0-698d7bc59-zr7ws" event={"ID":"3af50d6f-9968-40fd-9159-b6f7078023db","Type":"ContainerStarted","Data":"0c437352dfdf9d9d4f918afc7883bf279eb4964a09560cbd8ea29f956bca97bb"} Oct 30 17:47:42.510018 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:42.509994 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcdac0-698d7bc59-zr7ws" Oct 30 17:47:42.510242 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:42.510219 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcdac0-698d7bc59-zr7ws" event={"ID":"3af50d6f-9968-40fd-9159-b6f7078023db","Type":"ContainerStarted","Data":"e8af45149dc68249dce14c93e0c1b31af5e2ccfb25218acca0fd5b9a1361ab8b"} Oct 30 17:47:42.596133 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:42.592036 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcdac0-698d7bc59-zr7ws" podStartSLOduration=2.5920128719999997 podStartE2EDuration="2.592012872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:47:42.552358502 +0000 UTC m=+2120.604458434" watchObservedRunningTime="2026-10-30 17:47:42.592012872 +0000 UTC m=+2120.644112804" Oct 30 17:47:42.980885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:42.980313 4965 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 17:47:42.980885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:42.980382 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" podUID="f98b7bc7-2701-4a7e-9847-131fb50d65c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 17:47:43.482886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.482798 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2"] Oct 30 17:47:43.491293 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.491266 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:43.495359 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.495330 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2"] Oct 30 17:47:43.498439 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.498419 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-istio-envoy\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:43.498633 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.498617 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-workload-socket\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:43.498766 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.498755 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrqwl\" (UniqueName: \"kubernetes.io/projected/6c560829-549a-4f74-94c6-361c0662716b-kube-api-access-hrqwl\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:43.498902 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.498890 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c560829-549a-4f74-94c6-361c0662716b-istio-token\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:43.499035 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.499024 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-workload-certs\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:43.499186 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.499170 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-credential-socket\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:43.499389 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.499371 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-istio-data\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:43.499559 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.499544 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c560829-549a-4f74-94c6-361c0662716b-istiod-ca-cert\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:43.499758 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.499742 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c560829-549a-4f74-94c6-361c0662716b-istio-podinfo\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:43.601297 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.601260 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrqwl\" (UniqueName: \"kubernetes.io/projected/6c560829-549a-4f74-94c6-361c0662716b-kube-api-access-hrqwl\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:43.601834 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.601437 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c560829-549a-4f74-94c6-361c0662716b-istio-token\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:43.601834 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.601508 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-workload-certs\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:43.601834 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.601552 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-credential-socket\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:43.601834 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.601598 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-istio-data\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:43.601834 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.601683 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c560829-549a-4f74-94c6-361c0662716b-istiod-ca-cert\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:43.601834 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.601790 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c560829-549a-4f74-94c6-361c0662716b-istio-podinfo\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:43.602244 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.601985 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-istio-envoy\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:43.602244 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.602017 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-workload-socket\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:43.602244 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.602021 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-workload-certs\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:43.602244 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.602069 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-credential-socket\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:43.602498 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.602445 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-workload-socket\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:43.602543 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.602504 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c560829-549a-4f74-94c6-361c0662716b-istiod-ca-cert\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:43.602631 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.602611 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-istio-data\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:43.602729 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.602689 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-303ba5-openshift-default-dockercfg-d5wdq\"" Oct 30 17:47:43.604347 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.604321 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-istio-envoy\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:43.604635 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.604617 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c560829-549a-4f74-94c6-361c0662716b-istio-podinfo\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:43.624568 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.624534 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c560829-549a-4f74-94c6-361c0662716b-istio-token\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:43.624939 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.624918 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrqwl\" (UniqueName: \"kubernetes.io/projected/6c560829-549a-4f74-94c6-361c0662716b-kube-api-access-hrqwl\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:43.817271 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.817192 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:43.984523 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.983955 4965 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 17:47:43.984523 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:43.984018 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" podUID="f98b7bc7-2701-4a7e-9847-131fb50d65c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 17:47:44.172811 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:44.172782 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2"] Oct 30 17:47:44.175306 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:47:44.175274 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c560829_549a_4f74_94c6_361c0662716b.slice/crio-8012f320df6fd98c24de56b3521e5bf512b54d665953356011891af86fbd3eab WatchSource:0}: Error finding container 8012f320df6fd98c24de56b3521e5bf512b54d665953356011891af86fbd3eab: Status 404 returned error can't find the container with id 8012f320df6fd98c24de56b3521e5bf512b54d665953356011891af86fbd3eab Oct 30 17:47:44.179259 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:44.179222 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:47:44.179354 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:44.179298 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:47:44.179354 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:44.179331 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:47:44.536880 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:44.536770 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" event={"ID":"6c560829-549a-4f74-94c6-361c0662716b","Type":"ContainerStarted","Data":"3d46b129c731f4188dff5526f5df3deeb65137001d43d7f55494821423719b9a"} Oct 30 17:47:44.536880 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:44.536834 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" event={"ID":"6c560829-549a-4f74-94c6-361c0662716b","Type":"ContainerStarted","Data":"8012f320df6fd98c24de56b3521e5bf512b54d665953356011891af86fbd3eab"} Oct 30 17:47:44.614364 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:44.614298 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" podStartSLOduration=1.6142771919999999 podStartE2EDuration="1.614277192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:47:44.608992264 +0000 UTC m=+2122.661092220" watchObservedRunningTime="2026-10-30 17:47:44.614277192 +0000 UTC m=+2122.666377128" Oct 30 17:47:44.817351 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:44.817265 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:44.983627 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:44.983597 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:44.984319 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:44.984123 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:44.985432 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:44.985413 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:47:45.819923 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:45.819888 4965 patch_prober.go:29] interesting pod/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 17:47:45.820406 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:45.819963 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" podUID="6c560829-549a-4f74-94c6-361c0662716b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 17:47:45.844133 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:45.844099 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g"] Oct 30 17:47:45.849106 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:45.849082 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:45.880930 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:45.873367 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c596f1-openshift-default-dockercfg-zrcfh\"" Oct 30 17:47:45.911902 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:45.908673 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g"] Oct 30 17:47:45.944039 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:45.944003 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-workload-socket\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:45.944267 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:45.944086 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istio-envoy\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:45.944267 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:45.944203 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-workload-certs\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:45.944459 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:45.944327 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istio-podinfo\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:45.944459 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:45.944383 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istiod-ca-cert\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:45.944610 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:45.944503 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istio-data\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:45.944610 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:45.944542 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istio-token\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:45.944695 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:45.944673 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmjpq\" (UniqueName: \"kubernetes.io/projected/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-kube-api-access-lmjpq\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:45.944997 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:45.944972 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-credential-socket\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:46.049884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:46.049008 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-credential-socket\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:46.049884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:46.049130 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-workload-socket\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:46.049884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:46.049181 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istio-envoy\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:46.049884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:46.049259 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-workload-certs\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:46.049884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:46.049308 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istio-podinfo\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:46.049884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:46.049361 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istiod-ca-cert\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:46.049884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:46.049441 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istio-data\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:46.049884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:46.049477 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istio-token\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:46.049884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:46.049534 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmjpq\" (UniqueName: \"kubernetes.io/projected/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-kube-api-access-lmjpq\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:46.050533 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:46.050295 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-workload-certs\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:46.050597 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:46.050546 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-credential-socket\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:46.078938 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:46.050778 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-workload-socket\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:46.078938 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:46.073003 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istio-data\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:46.086886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:46.081504 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istio-podinfo\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:46.121884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:46.112368 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istiod-ca-cert\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:46.201926 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:46.176503 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istio-token\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:46.202220 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:46.176567 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istio-envoy\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:46.202360 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:46.185113 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmjpq\" (UniqueName: \"kubernetes.io/projected/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-kube-api-access-lmjpq\") pod \"e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:46.215233 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:46.215184 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:46.798397 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:47:46.798363 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8cbab33_ea35_4e2e_a82d_74e1f47c5b13.slice/crio-4114eaee609667222ad80169da93f19825fb1f20a33dbf87942d43f4d647514a WatchSource:0}: Error finding container 4114eaee609667222ad80169da93f19825fb1f20a33dbf87942d43f4d647514a: Status 404 returned error can't find the container with id 4114eaee609667222ad80169da93f19825fb1f20a33dbf87942d43f4d647514a Oct 30 17:47:46.798549 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:46.798411 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g"] Oct 30 17:47:46.801338 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:46.801303 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:47:46.801431 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:46.801369 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:47:46.801431 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:46.801415 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:47:46.818269 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:46.818215 4965 patch_prober.go:29] interesting pod/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 17:47:46.818419 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:46.818294 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" podUID="6c560829-549a-4f74-94c6-361c0662716b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 17:47:47.602128 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:47.602084 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" event={"ID":"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13","Type":"ContainerStarted","Data":"ff71ab324ee976af17b669afc245f2fc029c7e1dda2cde3f69189ac352495334"} Oct 30 17:47:47.602529 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:47.602181 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" event={"ID":"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13","Type":"ContainerStarted","Data":"4114eaee609667222ad80169da93f19825fb1f20a33dbf87942d43f4d647514a"} Oct 30 17:47:47.681885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:47.680495 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" podStartSLOduration=2.680474725 podStartE2EDuration="2.680474725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:47:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:47:47.679160632 +0000 UTC m=+2125.731260590" watchObservedRunningTime="2026-10-30 17:47:47.680474725 +0000 UTC m=+2125.732574656" Oct 30 17:47:47.819888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:47.819588 4965 patch_prober.go:29] interesting pod/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 17:47:47.819888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:47.819662 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" podUID="6c560829-549a-4f74-94c6-361c0662716b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 17:47:48.216548 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:48.216518 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:48.222567 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:48.222537 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:48.611278 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:48.611242 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:48.612309 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:48.612289 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:47:48.818211 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:48.818177 4965 patch_prober.go:29] interesting pod/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 17:47:48.818399 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:48.818256 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" podUID="6c560829-549a-4f74-94c6-361c0662716b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 17:47:49.827333 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:49.827297 4965 patch_prober.go:29] interesting pod/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 17:47:49.827989 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:49.827955 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" podUID="6c560829-549a-4f74-94c6-361c0662716b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 17:47:50.817528 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:50.817493 4965 patch_prober.go:29] interesting pod/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 17:47:50.817690 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:50.817546 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" podUID="6c560829-549a-4f74-94c6-361c0662716b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 17:47:51.663485 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.663448 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk"] Oct 30 17:47:51.668932 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.668909 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:47:51.703116 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.703091 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk"] Oct 30 17:47:51.805639 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.805598 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istio-envoy\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-nrfqk\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:47:51.805804 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.805712 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-workload-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-nrfqk\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:47:51.805804 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.805785 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istio-token\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-nrfqk\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:47:51.805992 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.805972 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-workload-certs\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-nrfqk\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:47:51.806042 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.806009 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-credential-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-nrfqk\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:47:51.806176 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.806158 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istio-podinfo\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-nrfqk\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:47:51.806329 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.806226 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istio-data\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-nrfqk\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:47:51.806329 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.806254 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istiod-ca-cert\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-nrfqk\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:47:51.806329 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.806281 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7f42\" (UniqueName: \"kubernetes.io/projected/21aa3bba-8be7-4766-b3ed-2c4684ccf614-kube-api-access-k7f42\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-nrfqk\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:47:51.823301 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.823278 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:51.823786 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.823765 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:51.824531 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.824516 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:47:51.908277 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.908239 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-workload-certs\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-nrfqk\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:47:51.908456 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.908298 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-credential-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-nrfqk\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:47:51.908456 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.908450 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istio-podinfo\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-nrfqk\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:47:51.908550 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.908526 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istio-data\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-nrfqk\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:47:51.908588 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.908567 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istiod-ca-cert\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-nrfqk\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:47:51.908623 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.908592 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7f42\" (UniqueName: \"kubernetes.io/projected/21aa3bba-8be7-4766-b3ed-2c4684ccf614-kube-api-access-k7f42\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-nrfqk\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:47:51.908694 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.908671 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istio-envoy\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-nrfqk\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:47:51.908824 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.908801 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-workload-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-nrfqk\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:47:51.908966 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.908880 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istio-token\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-nrfqk\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:47:51.909175 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.909145 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-credential-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-nrfqk\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:47:51.909380 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.909352 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-workload-certs\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-nrfqk\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:47:51.909682 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.909658 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istio-data\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-nrfqk\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:47:51.910501 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.910477 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-workload-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-nrfqk\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:47:51.910676 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.910552 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istiod-ca-cert\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-nrfqk\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:47:51.912384 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.912343 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istio-podinfo\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-nrfqk\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:47:51.912490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.912369 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istio-envoy\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-nrfqk\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:47:51.947826 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.947737 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istio-token\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-nrfqk\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:47:51.949004 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.948977 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7f42\" (UniqueName: \"kubernetes.io/projected/21aa3bba-8be7-4766-b3ed-2c4684ccf614-kube-api-access-k7f42\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-nrfqk\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:47:51.986107 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:51.986064 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:47:52.145109 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:52.145076 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c596f1-7c568c9f95-lgd2d"] Oct 30 17:47:52.151615 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:52.151587 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c596f1-7c568c9f95-lgd2d" Oct 30 17:47:52.167477 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:52.167429 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c596f1\"" Oct 30 17:47:52.167644 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:52.167523 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c596f1-dockercfg-kj585\"" Oct 30 17:47:52.168691 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:52.168668 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c596f1-7c568c9f95-lgd2d"] Oct 30 17:47:52.353379 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:52.282090 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cc9400e-90bd-4eb0-bf82-763acc3cf2c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c596f1-7c568c9f95-lgd2d\" (UID: \"4cc9400e-90bd-4eb0-bf82-763acc3cf2c7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c596f1-7c568c9f95-lgd2d" Oct 30 17:47:52.353379 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:52.282256 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwtmg\" (UniqueName: \"kubernetes.io/projected/4cc9400e-90bd-4eb0-bf82-763acc3cf2c7-kube-api-access-nwtmg\") pod \"payload-pre-processing-e2e-inf-b-c596f1-7c568c9f95-lgd2d\" (UID: \"4cc9400e-90bd-4eb0-bf82-763acc3cf2c7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c596f1-7c568c9f95-lgd2d" Oct 30 17:47:52.381747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:52.380927 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c596f1-778cb5cc94-glfgz"] Oct 30 17:47:52.390339 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:52.387209 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c596f1-778cb5cc94-glfgz" Oct 30 17:47:52.394719 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:52.394667 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c596f1-778cb5cc94-glfgz"] Oct 30 17:47:52.450888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:52.448839 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cc9400e-90bd-4eb0-bf82-763acc3cf2c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c596f1-7c568c9f95-lgd2d\" (UID: \"4cc9400e-90bd-4eb0-bf82-763acc3cf2c7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c596f1-7c568c9f95-lgd2d" Oct 30 17:47:52.450888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:52.449019 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwtmg\" (UniqueName: \"kubernetes.io/projected/4cc9400e-90bd-4eb0-bf82-763acc3cf2c7-kube-api-access-nwtmg\") pod \"payload-pre-processing-e2e-inf-b-c596f1-7c568c9f95-lgd2d\" (UID: \"4cc9400e-90bd-4eb0-bf82-763acc3cf2c7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c596f1-7c568c9f95-lgd2d" Oct 30 17:47:52.490084 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:52.488011 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cc9400e-90bd-4eb0-bf82-763acc3cf2c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c596f1-7c568c9f95-lgd2d\" (UID: \"4cc9400e-90bd-4eb0-bf82-763acc3cf2c7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c596f1-7c568c9f95-lgd2d" Oct 30 17:47:52.521294 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:52.519664 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:47:52.521294 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:52.519771 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:47:52.521294 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:52.519828 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:47:52.550775 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:52.550749 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwtmg\" (UniqueName: \"kubernetes.io/projected/4cc9400e-90bd-4eb0-bf82-763acc3cf2c7-kube-api-access-nwtmg\") pod \"payload-pre-processing-e2e-inf-b-c596f1-7c568c9f95-lgd2d\" (UID: \"4cc9400e-90bd-4eb0-bf82-763acc3cf2c7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c596f1-7c568c9f95-lgd2d" Oct 30 17:47:52.552135 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:52.552101 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wps8x\" (UniqueName: \"kubernetes.io/projected/e03a603f-5564-4340-ba6d-395abdf6fbd9-kube-api-access-wps8x\") pod \"payload-processing-e2e-inf-b-c596f1-778cb5cc94-glfgz\" (UID: \"e03a603f-5564-4340-ba6d-395abdf6fbd9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c596f1-778cb5cc94-glfgz" Oct 30 17:47:52.552499 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:52.552469 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e03a603f-5564-4340-ba6d-395abdf6fbd9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c596f1-778cb5cc94-glfgz\" (UID: \"e03a603f-5564-4340-ba6d-395abdf6fbd9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c596f1-778cb5cc94-glfgz" Oct 30 17:47:52.555513 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:52.555494 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk"] Oct 30 17:47:52.649332 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:52.649294 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" event={"ID":"21aa3bba-8be7-4766-b3ed-2c4684ccf614","Type":"ContainerStarted","Data":"62758005a1ef6a357a1a5c698d5e6fbdbceacf8e6b79add2877f6b88f8d0d7e4"} Oct 30 17:47:52.649493 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:52.649366 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" event={"ID":"21aa3bba-8be7-4766-b3ed-2c4684ccf614","Type":"ContainerStarted","Data":"e659db17d9460a5d6fac612338877dc788afa65e6eda135b0cb7f680ccc69c98"} Oct 30 17:47:52.654306 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:52.654269 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wps8x\" (UniqueName: \"kubernetes.io/projected/e03a603f-5564-4340-ba6d-395abdf6fbd9-kube-api-access-wps8x\") pod \"payload-processing-e2e-inf-b-c596f1-778cb5cc94-glfgz\" (UID: \"e03a603f-5564-4340-ba6d-395abdf6fbd9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c596f1-778cb5cc94-glfgz" Oct 30 17:47:52.654479 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:52.654457 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e03a603f-5564-4340-ba6d-395abdf6fbd9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c596f1-778cb5cc94-glfgz\" (UID: \"e03a603f-5564-4340-ba6d-395abdf6fbd9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c596f1-778cb5cc94-glfgz" Oct 30 17:47:52.655129 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:52.655104 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e03a603f-5564-4340-ba6d-395abdf6fbd9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c596f1-778cb5cc94-glfgz\" (UID: \"e03a603f-5564-4340-ba6d-395abdf6fbd9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c596f1-778cb5cc94-glfgz" Oct 30 17:47:52.677910 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:52.677848 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wps8x\" (UniqueName: \"kubernetes.io/projected/e03a603f-5564-4340-ba6d-395abdf6fbd9-kube-api-access-wps8x\") pod \"payload-processing-e2e-inf-b-c596f1-778cb5cc94-glfgz\" (UID: \"e03a603f-5564-4340-ba6d-395abdf6fbd9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c596f1-778cb5cc94-glfgz" Oct 30 17:47:52.697549 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:52.697497 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" podStartSLOduration=1.69748298 podStartE2EDuration="1.69748298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:47:52.69428326 +0000 UTC m=+2130.746383192" watchObservedRunningTime="2026-10-30 17:47:52.69748298 +0000 UTC m=+2130.749582911" Oct 30 17:47:52.772409 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:52.772375 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c596f1-7c568c9f95-lgd2d" Oct 30 17:47:52.856361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:52.856331 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c596f1-778cb5cc94-glfgz" Oct 30 17:47:52.987007 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:52.986972 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:47:53.085040 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.085007 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c596f1-7c568c9f95-lgd2d"] Oct 30 17:47:53.086468 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:47:53.086441 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc9400e_90bd_4eb0_bf82_763acc3cf2c7.slice/crio-098359e68d721edc7afb2dd62a078abd2f96dc6cc3afcd7dc867b491c7e4cd78 WatchSource:0}: Error finding container 098359e68d721edc7afb2dd62a078abd2f96dc6cc3afcd7dc867b491c7e4cd78: Status 404 returned error can't find the container with id 098359e68d721edc7afb2dd62a078abd2f96dc6cc3afcd7dc867b491c7e4cd78 Oct 30 17:47:53.189105 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:47:53.189072 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode03a603f_5564_4340_ba6d_395abdf6fbd9.slice/crio-1b388494893398ac725a1406f70a4aa201c57cab40186aa0d43bb3e0b8096de8 WatchSource:0}: Error finding container 1b388494893398ac725a1406f70a4aa201c57cab40186aa0d43bb3e0b8096de8: Status 404 returned error can't find the container with id 1b388494893398ac725a1406f70a4aa201c57cab40186aa0d43bb3e0b8096de8 Oct 30 17:47:53.192261 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.192218 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c596f1-778cb5cc94-glfgz"] Oct 30 17:47:53.194915 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.194842 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" podUID="a75d1f38-e7f2-47bd-b59a-538ee5b12412" containerName="main" containerID="cri-o://5dae539cd7fc6e802f9f059c421c37b5ab821f9b78421c395a923c9df4d9ed9b" gracePeriod=45 Oct 30 17:47:53.259194 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.259149 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" podUID="a75d1f38-e7f2-47bd-b59a-538ee5b12412" containerName="main" probeResult="failure" output="Get \"https://10.132.0.148:8000/ready\": dial tcp 10.132.0.148:8000: connect: connection refused" Oct 30 17:47:53.548415 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.548388 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" Oct 30 17:47:53.716530 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.711701 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a75d1f38-e7f2-47bd-b59a-538ee5b12412-tls-certs\" (UniqueName: \"kubernetes.io/secret/a75d1f38-e7f2-47bd-b59a-538ee5b12412-tls-certs\") pod \"a75d1f38-e7f2-47bd-b59a-538ee5b12412\" (UID: \"a75d1f38-e7f2-47bd-b59a-538ee5b12412\") " Oct 30 17:47:53.716530 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.711753 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a75d1f38-e7f2-47bd-b59a-538ee5b12412-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a75d1f38-e7f2-47bd-b59a-538ee5b12412-model-cache\") pod \"a75d1f38-e7f2-47bd-b59a-538ee5b12412\" (UID: \"a75d1f38-e7f2-47bd-b59a-538ee5b12412\") " Oct 30 17:47:53.716530 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.711824 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a75d1f38-e7f2-47bd-b59a-538ee5b12412-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a75d1f38-e7f2-47bd-b59a-538ee5b12412-dshm\") pod \"a75d1f38-e7f2-47bd-b59a-538ee5b12412\" (UID: \"a75d1f38-e7f2-47bd-b59a-538ee5b12412\") " Oct 30 17:47:53.721967 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.721931 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a75d1f38-e7f2-47bd-b59a-538ee5b12412-kube-api-access-rkfk7\" (UniqueName: \"kubernetes.io/projected/a75d1f38-e7f2-47bd-b59a-538ee5b12412-kube-api-access-rkfk7\") pod \"a75d1f38-e7f2-47bd-b59a-538ee5b12412\" (UID: \"a75d1f38-e7f2-47bd-b59a-538ee5b12412\") " Oct 30 17:47:53.722505 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.722480 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a75d1f38-e7f2-47bd-b59a-538ee5b12412-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a75d1f38-e7f2-47bd-b59a-538ee5b12412-tmp-dir\") pod \"a75d1f38-e7f2-47bd-b59a-538ee5b12412\" (UID: \"a75d1f38-e7f2-47bd-b59a-538ee5b12412\") " Oct 30 17:47:53.754890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.752677 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a75d1f38-e7f2-47bd-b59a-538ee5b12412-home\" (UniqueName: \"kubernetes.io/empty-dir/a75d1f38-e7f2-47bd-b59a-538ee5b12412-home\") pod \"a75d1f38-e7f2-47bd-b59a-538ee5b12412\" (UID: \"a75d1f38-e7f2-47bd-b59a-538ee5b12412\") " Oct 30 17:47:53.755126 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.749999 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a75d1f38-e7f2-47bd-b59a-538ee5b12412-tmp-dir" pod "a75d1f38-e7f2-47bd-b59a-538ee5b12412" (UID: "a75d1f38-e7f2-47bd-b59a-538ee5b12412"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:47:53.779092 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.777979 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a75d1f38-e7f2-47bd-b59a-538ee5b12412-home" pod "a75d1f38-e7f2-47bd-b59a-538ee5b12412" (UID: "a75d1f38-e7f2-47bd-b59a-538ee5b12412"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:47:53.779436 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.779401 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a75d1f38-e7f2-47bd-b59a-538ee5b12412-model-cache" pod "a75d1f38-e7f2-47bd-b59a-538ee5b12412" (UID: "a75d1f38-e7f2-47bd-b59a-538ee5b12412"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:47:53.865182 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.865141 4965 generic.go:356] "Generic (PLEG): container finished" podID="a75d1f38-e7f2-47bd-b59a-538ee5b12412" containerID="5dae539cd7fc6e802f9f059c421c37b5ab821f9b78421c395a923c9df4d9ed9b" exitCode=0 Oct 30 17:47:53.865396 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.865271 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" Oct 30 17:47:53.870965 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.866077 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" event={"ID":"a75d1f38-e7f2-47bd-b59a-538ee5b12412","Type":"ContainerDied","Data":"5dae539cd7fc6e802f9f059c421c37b5ab821f9b78421c395a923c9df4d9ed9b"} Oct 30 17:47:53.870965 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.866134 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v" event={"ID":"a75d1f38-e7f2-47bd-b59a-538ee5b12412","Type":"ContainerDied","Data":"14f357fe26380973d4d602a208dc0f2bf06f206ef049cee6531fbd083498f234"} Oct 30 17:47:53.870965 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.866158 4965 scope.go:122] "RemoveContainer" containerID="5dae539cd7fc6e802f9f059c421c37b5ab821f9b78421c395a923c9df4d9ed9b" Oct 30 17:47:53.871557 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.871482 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a75d1f38-e7f2-47bd-b59a-538ee5b12412-dshm" pod "a75d1f38-e7f2-47bd-b59a-538ee5b12412" (UID: "a75d1f38-e7f2-47bd-b59a-538ee5b12412"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:47:53.871838 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.871818 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a75d1f38-e7f2-47bd-b59a-538ee5b12412-tls-certs" pod "a75d1f38-e7f2-47bd-b59a-538ee5b12412" (UID: "a75d1f38-e7f2-47bd-b59a-538ee5b12412"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:47:53.876935 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.875495 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a75d1f38-e7f2-47bd-b59a-538ee5b12412-kube-api-access-rkfk7" pod "a75d1f38-e7f2-47bd-b59a-538ee5b12412" (UID: "a75d1f38-e7f2-47bd-b59a-538ee5b12412"). InnerVolumeSpecName "kube-api-access-rkfk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:47:53.884064 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.883394 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c596f1-778cb5cc94-glfgz" event={"ID":"e03a603f-5564-4340-ba6d-395abdf6fbd9","Type":"ContainerStarted","Data":"196b4dacdbac4935f666219b8b8e2ade2c5ff4a6ce26c0f3e0c727c87d9c7cfa"} Oct 30 17:47:53.884064 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.883453 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c596f1-778cb5cc94-glfgz" event={"ID":"e03a603f-5564-4340-ba6d-395abdf6fbd9","Type":"ContainerStarted","Data":"1b388494893398ac725a1406f70a4aa201c57cab40186aa0d43bb3e0b8096de8"} Oct 30 17:47:53.884706 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.884252 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c596f1-778cb5cc94-glfgz" Oct 30 17:47:53.887801 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.887342 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c596f1-7c568c9f95-lgd2d" event={"ID":"4cc9400e-90bd-4eb0-bf82-763acc3cf2c7","Type":"ContainerStarted","Data":"f3ec766c443dcb257462c73a1dfa419e8b78c81587f727c7a4c386bc0b294840"} Oct 30 17:47:53.887801 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.887390 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c596f1-7c568c9f95-lgd2d" event={"ID":"4cc9400e-90bd-4eb0-bf82-763acc3cf2c7","Type":"ContainerStarted","Data":"098359e68d721edc7afb2dd62a078abd2f96dc6cc3afcd7dc867b491c7e4cd78"} Oct 30 17:47:53.887801 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.887417 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c596f1-7c568c9f95-lgd2d" Oct 30 17:47:53.888796 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.888461 4965 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a75d1f38-e7f2-47bd-b59a-538ee5b12412-tls-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:53.888796 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.888577 4965 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a75d1f38-e7f2-47bd-b59a-538ee5b12412-model-cache\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:53.888796 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.888729 4965 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a75d1f38-e7f2-47bd-b59a-538ee5b12412-dshm\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:53.888796 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.888746 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkfk7\" (UniqueName: \"kubernetes.io/projected/a75d1f38-e7f2-47bd-b59a-538ee5b12412-kube-api-access-rkfk7\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:53.888796 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.888760 4965 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a75d1f38-e7f2-47bd-b59a-538ee5b12412-tmp-dir\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:53.888796 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.888775 4965 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a75d1f38-e7f2-47bd-b59a-538ee5b12412-home\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:47:53.958662 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.958637 4965 scope.go:122] "RemoveContainer" containerID="5dae539cd7fc6e802f9f059c421c37b5ab821f9b78421c395a923c9df4d9ed9b" Oct 30 17:47:53.971030 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:47:53.970023 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dae539cd7fc6e802f9f059c421c37b5ab821f9b78421c395a923c9df4d9ed9b\": container with ID starting with 5dae539cd7fc6e802f9f059c421c37b5ab821f9b78421c395a923c9df4d9ed9b not found: ID does not exist" containerID="5dae539cd7fc6e802f9f059c421c37b5ab821f9b78421c395a923c9df4d9ed9b" Oct 30 17:47:53.971030 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.970070 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dae539cd7fc6e802f9f059c421c37b5ab821f9b78421c395a923c9df4d9ed9b"} err="failed to get container status \"5dae539cd7fc6e802f9f059c421c37b5ab821f9b78421c395a923c9df4d9ed9b\": rpc error: code = NotFound desc = could not find container \"5dae539cd7fc6e802f9f059c421c37b5ab821f9b78421c395a923c9df4d9ed9b\": container with ID starting with 5dae539cd7fc6e802f9f059c421c37b5ab821f9b78421c395a923c9df4d9ed9b not found: ID does not exist" Oct 30 17:47:53.997299 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.997266 4965 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 17:47:53.998166 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:53.998133 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" podUID="21aa3bba-8be7-4766-b3ed-2c4684ccf614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 17:47:54.000064 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:54.000018 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c596f1-7c568c9f95-lgd2d" podStartSLOduration=2.000001348 podStartE2EDuration="2.000001348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:47:53.99772625 +0000 UTC m=+2132.049826205" watchObservedRunningTime="2026-10-30 17:47:54.000001348 +0000 UTC m=+2132.052101307" Oct 30 17:47:54.002651 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:54.002607 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-c596f1-778cb5cc94-glfgz" podStartSLOduration=2.002596404 podStartE2EDuration="2.002596404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:47:53.952879581 +0000 UTC m=+2132.004979544" watchObservedRunningTime="2026-10-30 17:47:54.002596404 +0000 UTC m=+2132.054696339" Oct 30 17:47:54.269774 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:54.269628 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v"] Oct 30 17:47:54.275056 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:54.275019 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-143041/test-model-143041-kserve-568689b445-5kz6v"] Oct 30 17:47:54.549183 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:54.549107 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcdac0-698d7bc59-zr7ws" Oct 30 17:47:54.987459 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:54.987422 4965 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 17:47:54.987888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:54.987474 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" podUID="21aa3bba-8be7-4766-b3ed-2c4684ccf614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 17:47:55.987443 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:55.987404 4965 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 17:47:55.987634 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:55.987460 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" podUID="21aa3bba-8be7-4766-b3ed-2c4684ccf614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 17:47:56.101214 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:56.101183 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a75d1f38-e7f2-47bd-b59a-538ee5b12412" path="/var/lib/kubelet/pods/a75d1f38-e7f2-47bd-b59a-538ee5b12412/volumes" Oct 30 17:47:56.987551 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:56.987513 4965 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 17:47:56.987757 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:56.987575 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" podUID="21aa3bba-8be7-4766-b3ed-2c4684ccf614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 17:47:57.788965 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.788929 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45"] Oct 30 17:47:57.789848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.789828 4965 cpu_manager.go:414] "containerMap: removing container" podUID="a75d1f38-e7f2-47bd-b59a-538ee5b12412" containerName="main" Oct 30 17:47:57.789848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.789847 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a75d1f38-e7f2-47bd-b59a-538ee5b12412" containerName="main" Oct 30 17:47:57.790030 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.789984 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="a75d1f38-e7f2-47bd-b59a-538ee5b12412" containerName="main" Oct 30 17:47:57.795510 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.795487 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:47:57.800609 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.800579 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45"] Oct 30 17:47:57.810654 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.810627 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-38f445-openshift-default-dockercfg-x7t6p\"" Oct 30 17:47:57.843621 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.843582 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e75131de-9643-4578-9a0a-e5c41d409131-istio-podinfo\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:47:57.843787 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.843719 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-workload-socket\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:47:57.843844 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.843802 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-workload-certs\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:47:57.843959 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.843936 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-istio-envoy\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:47:57.844008 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.843992 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4j4l\" (UniqueName: \"kubernetes.io/projected/e75131de-9643-4578-9a0a-e5c41d409131-kube-api-access-c4j4l\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:47:57.844052 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.844030 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-istio-data\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:47:57.844158 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.844138 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e75131de-9643-4578-9a0a-e5c41d409131-istiod-ca-cert\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:47:57.844338 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.844324 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e75131de-9643-4578-9a0a-e5c41d409131-istio-token\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:47:57.844397 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.844381 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-credential-socket\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:47:57.946339 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.946299 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-credential-socket\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:47:57.946520 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.946350 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e75131de-9643-4578-9a0a-e5c41d409131-istio-podinfo\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:47:57.946520 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.946410 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-workload-socket\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:47:57.946520 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.946478 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-workload-certs\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:47:57.946520 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.946506 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-istio-envoy\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:47:57.946769 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.946547 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4j4l\" (UniqueName: \"kubernetes.io/projected/e75131de-9643-4578-9a0a-e5c41d409131-kube-api-access-c4j4l\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:47:57.946769 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.946609 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-istio-data\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:47:57.946912 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.946772 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-credential-socket\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:47:57.946912 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.946791 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-workload-socket\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:47:57.946912 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.946853 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-workload-certs\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:47:57.947103 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.947055 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-istio-data\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:47:57.947167 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.947145 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e75131de-9643-4578-9a0a-e5c41d409131-istiod-ca-cert\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:47:57.947370 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.947353 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e75131de-9643-4578-9a0a-e5c41d409131-istio-token\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:47:57.947917 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.947858 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e75131de-9643-4578-9a0a-e5c41d409131-istiod-ca-cert\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:47:57.948949 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.948916 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-istio-envoy\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:47:57.949072 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.949043 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e75131de-9643-4578-9a0a-e5c41d409131-istio-podinfo\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:47:57.985885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.985359 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e75131de-9643-4578-9a0a-e5c41d409131-istio-token\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:47:57.985885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.985649 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4j4l\" (UniqueName: \"kubernetes.io/projected/e75131de-9643-4578-9a0a-e5c41d409131-kube-api-access-c4j4l\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:47:57.987147 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.987047 4965 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 17:47:57.987147 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:57.987108 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" podUID="21aa3bba-8be7-4766-b3ed-2c4684ccf614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 17:47:58.113735 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:58.113706 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:47:58.488068 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:47:58.488033 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode75131de_9643_4578_9a0a_e5c41d409131.slice/crio-2acecfc27934ed610edb11f96cb0255a922f71d822de5b77ba21f952df19702e WatchSource:0}: Error finding container 2acecfc27934ed610edb11f96cb0255a922f71d822de5b77ba21f952df19702e: Status 404 returned error can't find the container with id 2acecfc27934ed610edb11f96cb0255a922f71d822de5b77ba21f952df19702e Oct 30 17:47:58.488210 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:58.488186 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45"] Oct 30 17:47:58.491162 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:58.491129 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:47:58.491238 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:58.491203 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:47:58.491286 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:58.491238 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:47:58.965883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:58.956382 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" event={"ID":"e75131de-9643-4578-9a0a-e5c41d409131","Type":"ContainerStarted","Data":"664fc534a0f87d9f7871e86fcf1e33be226c2366598d850f3df7d36b2a3e31b5"} Oct 30 17:47:58.965883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:58.956456 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" event={"ID":"e75131de-9643-4578-9a0a-e5c41d409131","Type":"ContainerStarted","Data":"2acecfc27934ed610edb11f96cb0255a922f71d822de5b77ba21f952df19702e"} Oct 30 17:47:58.987884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:58.987430 4965 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 17:47:58.987884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:58.987490 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" podUID="21aa3bba-8be7-4766-b3ed-2c4684ccf614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 17:47:59.013347 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:59.013276 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" podStartSLOduration=2.013256396 podStartE2EDuration="2.013256396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:47:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:47:59.008008571 +0000 UTC m=+2137.060108528" watchObservedRunningTime="2026-10-30 17:47:59.013256396 +0000 UTC m=+2137.065356328" Oct 30 17:47:59.114765 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:59.114731 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:47:59.116478 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:59.116452 4965 patch_prober.go:29] interesting pod/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 17:47:59.116594 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:59.116498 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" podUID="e75131de-9643-4578-9a0a-e5c41d409131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 17:47:59.995396 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:59.995367 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:47:59.995839 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:59.995800 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:47:59.996603 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:47:59.996582 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:48:00.126350 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:00.126305 4965 patch_prober.go:29] interesting pod/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 17:48:00.126843 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:00.126485 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" podUID="e75131de-9643-4578-9a0a-e5c41d409131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 17:48:01.114586 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:01.114548 4965 patch_prober.go:29] interesting pod/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 17:48:01.114810 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:01.114605 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" podUID="e75131de-9643-4578-9a0a-e5c41d409131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 17:48:02.114100 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:02.114064 4965 patch_prober.go:29] interesting pod/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 17:48:02.114481 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:02.114114 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" podUID="e75131de-9643-4578-9a0a-e5c41d409131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 17:48:03.114578 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:03.114344 4965 patch_prober.go:29] interesting pod/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 17:48:03.114578 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:03.114422 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" podUID="e75131de-9643-4578-9a0a-e5c41d409131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 17:48:04.118918 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:04.118882 4965 patch_prober.go:29] interesting pod/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 17:48:04.120492 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:04.120450 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" podUID="e75131de-9643-4578-9a0a-e5c41d409131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 17:48:04.901833 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:04.901799 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c596f1-7c568c9f95-lgd2d" Oct 30 17:48:04.903268 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:04.901934 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c596f1-778cb5cc94-glfgz" Oct 30 17:48:05.235031 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:05.234925 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:48:05.236214 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:05.236125 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:48:05.236881 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:05.236846 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:48:05.997923 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:48:05.997889 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:48:06.683803 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:06.683740 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz"] Oct 30 17:48:06.696287 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:06.696256 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:06.723527 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:06.723494 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz"] Oct 30 17:48:06.784762 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:06.784723 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrpnm\" (UniqueName: \"kubernetes.io/projected/93dead59-30a5-4adb-a6f1-e54a36ff320f-kube-api-access-wrpnm\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-fw7dz\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:06.784943 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:06.784780 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93dead59-30a5-4adb-a6f1-e54a36ff320f-istio-podinfo\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-fw7dz\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:06.784943 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:06.784857 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-workload-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-fw7dz\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:06.784943 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:06.784937 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-istio-envoy\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-fw7dz\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:06.785088 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:06.784996 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-workload-certs\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-fw7dz\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:06.785088 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:06.785083 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-istio-data\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-fw7dz\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:06.785224 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:06.785203 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93dead59-30a5-4adb-a6f1-e54a36ff320f-istiod-ca-cert\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-fw7dz\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:06.785288 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:06.785268 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93dead59-30a5-4adb-a6f1-e54a36ff320f-istio-token\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-fw7dz\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:06.785447 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:06.785434 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-credential-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-fw7dz\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:06.886763 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:06.886726 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93dead59-30a5-4adb-a6f1-e54a36ff320f-istiod-ca-cert\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-fw7dz\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:06.886965 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:06.886790 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93dead59-30a5-4adb-a6f1-e54a36ff320f-istio-token\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-fw7dz\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:06.887042 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:06.887024 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-credential-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-fw7dz\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:06.887107 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:06.887068 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrpnm\" (UniqueName: \"kubernetes.io/projected/93dead59-30a5-4adb-a6f1-e54a36ff320f-kube-api-access-wrpnm\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-fw7dz\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:06.887170 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:06.887113 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93dead59-30a5-4adb-a6f1-e54a36ff320f-istio-podinfo\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-fw7dz\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:06.887170 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:06.887149 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-workload-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-fw7dz\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:06.887270 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:06.887230 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-istio-envoy\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-fw7dz\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:06.887329 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:06.887276 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-workload-certs\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-fw7dz\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:06.887475 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:06.887400 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-credential-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-fw7dz\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:06.887475 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:06.887405 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-istio-data\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-fw7dz\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:06.887618 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:06.887549 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-workload-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-fw7dz\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:06.887618 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:06.887550 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93dead59-30a5-4adb-a6f1-e54a36ff320f-istiod-ca-cert\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-fw7dz\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:06.887749 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:06.887728 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-istio-data\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-fw7dz\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:06.887808 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:06.887756 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-workload-certs\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-fw7dz\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:06.889483 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:06.889462 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-istio-envoy\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-fw7dz\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:06.889783 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:06.889762 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93dead59-30a5-4adb-a6f1-e54a36ff320f-istio-podinfo\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-fw7dz\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:06.910196 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:06.910171 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93dead59-30a5-4adb-a6f1-e54a36ff320f-istio-token\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-fw7dz\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:06.910349 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:06.910330 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrpnm\" (UniqueName: \"kubernetes.io/projected/93dead59-30a5-4adb-a6f1-e54a36ff320f-kube-api-access-wrpnm\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-fw7dz\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:07.035332 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:07.035245 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:07.433514 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:48:07.433482 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93dead59_30a5_4adb_a6f1_e54a36ff320f.slice/crio-9b567489ea643fc298c2bd7aa76be6d1cf8261b5c95d37344b16bd9dedc0e8b4 WatchSource:0}: Error finding container 9b567489ea643fc298c2bd7aa76be6d1cf8261b5c95d37344b16bd9dedc0e8b4: Status 404 returned error can't find the container with id 9b567489ea643fc298c2bd7aa76be6d1cf8261b5c95d37344b16bd9dedc0e8b4 Oct 30 17:48:07.435174 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:07.435150 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz"] Oct 30 17:48:07.437186 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:07.437150 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:48:07.437263 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:07.437221 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:48:07.437263 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:07.437250 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:48:08.133640 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:08.133600 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" event={"ID":"93dead59-30a5-4adb-a6f1-e54a36ff320f","Type":"ContainerStarted","Data":"50a1b36a7c53a2eb33152db0525f23b8a901458853add1f4accec2ffe9b15378"} Oct 30 17:48:08.134090 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:08.133654 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" event={"ID":"93dead59-30a5-4adb-a6f1-e54a36ff320f","Type":"ContainerStarted","Data":"9b567489ea643fc298c2bd7aa76be6d1cf8261b5c95d37344b16bd9dedc0e8b4"} Oct 30 17:48:08.196591 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:08.196532 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" podStartSLOduration=2.196518479 podStartE2EDuration="2.196518479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:48:08.193064943 +0000 UTC m=+2146.245164894" watchObservedRunningTime="2026-10-30 17:48:08.196518479 +0000 UTC m=+2146.248618409" Oct 30 17:48:09.035924 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:09.035858 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:09.037793 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:09.037766 4965 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 17:48:09.037939 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:09.037813 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" podUID="93dead59-30a5-4adb-a6f1-e54a36ff320f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 17:48:10.035773 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:10.035734 4965 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 17:48:10.036183 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:10.035791 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" podUID="93dead59-30a5-4adb-a6f1-e54a36ff320f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 17:48:11.036497 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:11.036466 4965 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 17:48:11.036940 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:11.036514 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" podUID="93dead59-30a5-4adb-a6f1-e54a36ff320f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 17:48:12.038887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:12.035763 4965 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 17:48:12.038887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:12.035841 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" podUID="93dead59-30a5-4adb-a6f1-e54a36ff320f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 17:48:13.036013 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:13.035973 4965 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 17:48:13.036202 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:13.036029 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" podUID="93dead59-30a5-4adb-a6f1-e54a36ff320f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 17:48:14.040982 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:14.040951 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:14.041469 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:14.041454 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:14.042278 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:14.042248 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:48:15.942921 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:15.942884 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9"] Oct 30 17:48:15.943443 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:15.943241 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9" podUID="5a0c4b63-604c-49de-9f33-4465aec216bc" containerName="maas-api" containerID="cri-o://4c9f8b30ee25ac24925ed69fe7c6bcebe926e12c41a338835b71fd0e58a3748c" gracePeriod=30 Oct 30 17:48:16.330325 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:16.330281 4965 generic.go:356] "Generic (PLEG): container finished" podID="5a0c4b63-604c-49de-9f33-4465aec216bc" containerID="4c9f8b30ee25ac24925ed69fe7c6bcebe926e12c41a338835b71fd0e58a3748c" exitCode=0 Oct 30 17:48:16.330529 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:16.330454 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9" event={"ID":"5a0c4b63-604c-49de-9f33-4465aec216bc","Type":"ContainerDied","Data":"4c9f8b30ee25ac24925ed69fe7c6bcebe926e12c41a338835b71fd0e58a3748c"} Oct 30 17:48:16.330529 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:16.330505 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9" event={"ID":"5a0c4b63-604c-49de-9f33-4465aec216bc","Type":"ContainerDied","Data":"bb9e5104e296298477d59186810d4e4453bdec90259e9598457b77d8a38dd1b7"} Oct 30 17:48:16.330529 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:16.330530 4965 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb9e5104e296298477d59186810d4e4453bdec90259e9598457b77d8a38dd1b7" Oct 30 17:48:16.330737 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:16.330548 4965 scope.go:122] "RemoveContainer" containerID="f7ebed19cc30a426b2e9bf305e85cb35e8eb98cf115c5c50970aa5a5e02b0198" Oct 30 17:48:16.332477 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:16.332455 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9" Oct 30 17:48:16.400638 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:16.400595 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a0c4b63-604c-49de-9f33-4465aec216bc-kube-api-access-ns6zw\" (UniqueName: \"kubernetes.io/projected/5a0c4b63-604c-49de-9f33-4465aec216bc-kube-api-access-ns6zw\") pod \"5a0c4b63-604c-49de-9f33-4465aec216bc\" (UID: \"5a0c4b63-604c-49de-9f33-4465aec216bc\") " Oct 30 17:48:16.401831 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:16.401804 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a0c4b63-604c-49de-9f33-4465aec216bc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a0c4b63-604c-49de-9f33-4465aec216bc-metrics-certs\") pod \"5a0c4b63-604c-49de-9f33-4465aec216bc\" (UID: \"5a0c4b63-604c-49de-9f33-4465aec216bc\") " Oct 30 17:48:16.402378 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:16.402351 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a0c4b63-604c-49de-9f33-4465aec216bc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0c4b63-604c-49de-9f33-4465aec216bc-maas-api-tls\") pod \"5a0c4b63-604c-49de-9f33-4465aec216bc\" (UID: \"5a0c4b63-604c-49de-9f33-4465aec216bc\") " Oct 30 17:48:16.402553 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:16.402535 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a0c4b63-604c-49de-9f33-4465aec216bc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a0c4b63-604c-49de-9f33-4465aec216bc-openshift-service-ca\") pod \"5a0c4b63-604c-49de-9f33-4465aec216bc\" (UID: \"5a0c4b63-604c-49de-9f33-4465aec216bc\") " Oct 30 17:48:16.442345 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:16.422300 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a0c4b63-604c-49de-9f33-4465aec216bc-metrics-certs" pod "5a0c4b63-604c-49de-9f33-4465aec216bc" (UID: "5a0c4b63-604c-49de-9f33-4465aec216bc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:48:16.460913 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:16.447208 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a0c4b63-604c-49de-9f33-4465aec216bc-openshift-service-ca" pod "5a0c4b63-604c-49de-9f33-4465aec216bc" (UID: "5a0c4b63-604c-49de-9f33-4465aec216bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:48:16.483634 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:16.483526 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a0c4b63-604c-49de-9f33-4465aec216bc-kube-api-access-ns6zw" pod "5a0c4b63-604c-49de-9f33-4465aec216bc" (UID: "5a0c4b63-604c-49de-9f33-4465aec216bc"). InnerVolumeSpecName "kube-api-access-ns6zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:48:16.507420 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:16.507386 4965 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a0c4b63-604c-49de-9f33-4465aec216bc-openshift-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:48:16.507683 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:16.507662 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns6zw\" (UniqueName: \"kubernetes.io/projected/5a0c4b63-604c-49de-9f33-4465aec216bc-kube-api-access-ns6zw\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:48:16.507802 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:16.507787 4965 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a0c4b63-604c-49de-9f33-4465aec216bc-metrics-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:48:16.592884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:16.592004 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a0c4b63-604c-49de-9f33-4465aec216bc-maas-api-tls" pod "5a0c4b63-604c-49de-9f33-4465aec216bc" (UID: "5a0c4b63-604c-49de-9f33-4465aec216bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:48:16.905093 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:16.905016 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0c4b63-604c-49de-9f33-4465aec216bc-maas-api-tls\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:48:17.340654 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:17.340622 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9" Oct 30 17:48:17.422168 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:17.422139 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9"] Oct 30 17:48:17.428127 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:17.428102 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bba90b-5d8b669b85-s5xs9"] Oct 30 17:48:18.100803 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:18.100768 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a0c4b63-604c-49de-9f33-4465aec216bc" path="/var/lib/kubelet/pods/5a0c4b63-604c-49de-9f33-4465aec216bc/volumes" Oct 30 17:48:24.658254 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:24.658219 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs"] Oct 30 17:48:24.658744 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:24.658476 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" podUID="81aec189-33b7-48e3-a558-65938570fc4f" containerName="istio-proxy" containerID="cri-o://fd97f08d23fdd5ee2c8004b0f9ffe3adc0783bf540d2bb65c24bd0f471175a6b" gracePeriod=30 Oct 30 17:48:25.793202 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:25.793172 4965 scope.go:122] "RemoveContainer" containerID="4c9f8b30ee25ac24925ed69fe7c6bcebe926e12c41a338835b71fd0e58a3748c" Oct 30 17:48:27.798102 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:27.798069 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:48:30.526605 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.526569 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:48:30.529840 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.529814 4965 generic.go:356] "Generic (PLEG): container finished" podID="81aec189-33b7-48e3-a558-65938570fc4f" containerID="fd97f08d23fdd5ee2c8004b0f9ffe3adc0783bf540d2bb65c24bd0f471175a6b" exitCode=0 Oct 30 17:48:30.530014 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.529978 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" event={"ID":"81aec189-33b7-48e3-a558-65938570fc4f","Type":"ContainerDied","Data":"fd97f08d23fdd5ee2c8004b0f9ffe3adc0783bf540d2bb65c24bd0f471175a6b"} Oct 30 17:48:30.530106 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.530023 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" event={"ID":"81aec189-33b7-48e3-a558-65938570fc4f","Type":"ContainerDied","Data":"58ba539b596ab93d066a57be2720edc877112a5e292b43129f696c4a2d606133"} Oct 30 17:48:30.530106 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.530046 4965 scope.go:122] "RemoveContainer" containerID="fd97f08d23fdd5ee2c8004b0f9ffe3adc0783bf540d2bb65c24bd0f471175a6b" Oct 30 17:48:30.530233 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.530176 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs" Oct 30 17:48:30.544199 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.544182 4965 scope.go:122] "RemoveContainer" containerID="fd97f08d23fdd5ee2c8004b0f9ffe3adc0783bf540d2bb65c24bd0f471175a6b" Oct 30 17:48:30.544615 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:48:30.544587 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd97f08d23fdd5ee2c8004b0f9ffe3adc0783bf540d2bb65c24bd0f471175a6b\": container with ID starting with fd97f08d23fdd5ee2c8004b0f9ffe3adc0783bf540d2bb65c24bd0f471175a6b not found: ID does not exist" containerID="fd97f08d23fdd5ee2c8004b0f9ffe3adc0783bf540d2bb65c24bd0f471175a6b" Oct 30 17:48:30.544686 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.544628 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd97f08d23fdd5ee2c8004b0f9ffe3adc0783bf540d2bb65c24bd0f471175a6b"} err="failed to get container status \"fd97f08d23fdd5ee2c8004b0f9ffe3adc0783bf540d2bb65c24bd0f471175a6b\": rpc error: code = NotFound desc = could not find container \"fd97f08d23fdd5ee2c8004b0f9ffe3adc0783bf540d2bb65c24bd0f471175a6b\": container with ID starting with fd97f08d23fdd5ee2c8004b0f9ffe3adc0783bf540d2bb65c24bd0f471175a6b not found: ID does not exist" Oct 30 17:48:30.691584 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.691549 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-workload-socket\") pod \"81aec189-33b7-48e3-a558-65938570fc4f\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " Oct 30 17:48:30.691760 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.691638 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81aec189-33b7-48e3-a558-65938570fc4f-kube-api-access-pqgpl\" (UniqueName: \"kubernetes.io/projected/81aec189-33b7-48e3-a558-65938570fc4f-kube-api-access-pqgpl\") pod \"81aec189-33b7-48e3-a558-65938570fc4f\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " Oct 30 17:48:30.691760 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.691694 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-workload-certs\") pod \"81aec189-33b7-48e3-a558-65938570fc4f\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " Oct 30 17:48:30.691909 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.691781 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-istio-envoy\") pod \"81aec189-33b7-48e3-a558-65938570fc4f\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " Oct 30 17:48:30.691909 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.691829 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-istio-data\") pod \"81aec189-33b7-48e3-a558-65938570fc4f\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " Oct 30 17:48:30.691909 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.691895 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-workload-socket" pod "81aec189-33b7-48e3-a558-65938570fc4f" (UID: "81aec189-33b7-48e3-a558-65938570fc4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:48:30.692082 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.692058 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-credential-socket\") pod \"81aec189-33b7-48e3-a558-65938570fc4f\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " Oct 30 17:48:30.692147 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.692094 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-istio-data" pod "81aec189-33b7-48e3-a558-65938570fc4f" (UID: "81aec189-33b7-48e3-a558-65938570fc4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:48:30.692147 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.692114 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-workload-certs" pod "81aec189-33b7-48e3-a558-65938570fc4f" (UID: "81aec189-33b7-48e3-a558-65938570fc4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:48:30.692147 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.692122 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81aec189-33b7-48e3-a558-65938570fc4f-istio-token\" (UniqueName: \"kubernetes.io/projected/81aec189-33b7-48e3-a558-65938570fc4f-istio-token\") pod \"81aec189-33b7-48e3-a558-65938570fc4f\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " Oct 30 17:48:30.693458 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.692225 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81aec189-33b7-48e3-a558-65938570fc4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81aec189-33b7-48e3-a558-65938570fc4f-istiod-ca-cert\") pod \"81aec189-33b7-48e3-a558-65938570fc4f\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " Oct 30 17:48:30.693458 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.692321 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-credential-socket" pod "81aec189-33b7-48e3-a558-65938570fc4f" (UID: "81aec189-33b7-48e3-a558-65938570fc4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:48:30.693458 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.692368 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81aec189-33b7-48e3-a558-65938570fc4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81aec189-33b7-48e3-a558-65938570fc4f-istio-podinfo\") pod \"81aec189-33b7-48e3-a558-65938570fc4f\" (UID: \"81aec189-33b7-48e3-a558-65938570fc4f\") " Oct 30 17:48:30.693458 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.692624 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81aec189-33b7-48e3-a558-65938570fc4f-istiod-ca-cert" pod "81aec189-33b7-48e3-a558-65938570fc4f" (UID: "81aec189-33b7-48e3-a558-65938570fc4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:48:30.693458 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.693364 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:48:30.693458 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.693389 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81aec189-33b7-48e3-a558-65938570fc4f-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:48:30.693458 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.693405 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:48:30.693458 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.693419 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:48:30.693458 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.693432 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:48:30.694779 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.694733 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81aec189-33b7-48e3-a558-65938570fc4f-kube-api-access-pqgpl" pod "81aec189-33b7-48e3-a558-65938570fc4f" (UID: "81aec189-33b7-48e3-a558-65938570fc4f"). InnerVolumeSpecName "kube-api-access-pqgpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:48:30.694954 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.694834 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81aec189-33b7-48e3-a558-65938570fc4f-istio-token" pod "81aec189-33b7-48e3-a558-65938570fc4f" (UID: "81aec189-33b7-48e3-a558-65938570fc4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:48:30.695124 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.695096 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-istio-envoy" pod "81aec189-33b7-48e3-a558-65938570fc4f" (UID: "81aec189-33b7-48e3-a558-65938570fc4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:48:30.695408 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.695386 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81aec189-33b7-48e3-a558-65938570fc4f-istio-podinfo" pod "81aec189-33b7-48e3-a558-65938570fc4f" (UID: "81aec189-33b7-48e3-a558-65938570fc4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:48:30.794713 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.794625 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81aec189-33b7-48e3-a558-65938570fc4f-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:48:30.794713 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.794654 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqgpl\" (UniqueName: \"kubernetes.io/projected/81aec189-33b7-48e3-a558-65938570fc4f-kube-api-access-pqgpl\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:48:30.794713 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.794665 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81aec189-33b7-48e3-a558-65938570fc4f-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:48:30.794713 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.794675 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81aec189-33b7-48e3-a558-65938570fc4f-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:48:30.885345 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.885308 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs"] Oct 30 17:48:30.894913 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:30.894851 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-qjfxs"] Oct 30 17:48:32.101966 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:32.101934 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81aec189-33b7-48e3-a558-65938570fc4f" path="/var/lib/kubelet/pods/81aec189-33b7-48e3-a558-65938570fc4f/volumes" Oct 30 17:48:35.998461 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:48:35.998419 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:48:36.503727 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:36.503693 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-143041-6bfddb8867-5tnjk"] Oct 30 17:48:40.159875 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:40.159836 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962"] Oct 30 17:48:40.162535 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:40.160675 4965 cpu_manager.go:414] "containerMap: removing container" podUID="81aec189-33b7-48e3-a558-65938570fc4f" containerName="istio-proxy" Oct 30 17:48:40.162535 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:40.160692 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81aec189-33b7-48e3-a558-65938570fc4f" containerName="istio-proxy" Oct 30 17:48:40.162535 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:40.160703 4965 cpu_manager.go:414] "containerMap: removing container" podUID="5a0c4b63-604c-49de-9f33-4465aec216bc" containerName="maas-api" Oct 30 17:48:40.162535 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:40.160708 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a0c4b63-604c-49de-9f33-4465aec216bc" containerName="maas-api" Oct 30 17:48:40.162535 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:40.160716 4965 cpu_manager.go:414] "containerMap: removing container" podUID="5a0c4b63-604c-49de-9f33-4465aec216bc" containerName="maas-api" Oct 30 17:48:40.162535 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:40.160721 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a0c4b63-604c-49de-9f33-4465aec216bc" containerName="maas-api" Oct 30 17:48:40.162535 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:40.160852 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="81aec189-33b7-48e3-a558-65938570fc4f" containerName="istio-proxy" Oct 30 17:48:40.162535 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:40.160879 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a0c4b63-604c-49de-9f33-4465aec216bc" containerName="maas-api" Oct 30 17:48:40.164517 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:40.164501 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962" Oct 30 17:48:40.179304 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:40.179280 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-c596f1-serving-cert\"" Oct 30 17:48:40.199024 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:40.198988 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962"] Oct 30 17:48:40.220504 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:40.220472 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a0b9f7e-5cae-4e6d-9737-6aef893fda01-metrics-certs\") pod \"maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962\" (UID: \"3a0b9f7e-5cae-4e6d-9737-6aef893fda01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962" Oct 30 17:48:40.220725 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:40.220699 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtjn4\" (UniqueName: \"kubernetes.io/projected/3a0b9f7e-5cae-4e6d-9737-6aef893fda01-kube-api-access-qtjn4\") pod \"maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962\" (UID: \"3a0b9f7e-5cae-4e6d-9737-6aef893fda01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962" Oct 30 17:48:40.220823 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:40.220808 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a0b9f7e-5cae-4e6d-9737-6aef893fda01-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962\" (UID: \"3a0b9f7e-5cae-4e6d-9737-6aef893fda01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962" Oct 30 17:48:40.221016 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:40.220993 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a0b9f7e-5cae-4e6d-9737-6aef893fda01-maas-api-tls\") pod \"maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962\" (UID: \"3a0b9f7e-5cae-4e6d-9737-6aef893fda01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962" Oct 30 17:48:40.322179 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:40.322146 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a0b9f7e-5cae-4e6d-9737-6aef893fda01-metrics-certs\") pod \"maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962\" (UID: \"3a0b9f7e-5cae-4e6d-9737-6aef893fda01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962" Oct 30 17:48:40.322350 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:40.322263 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtjn4\" (UniqueName: \"kubernetes.io/projected/3a0b9f7e-5cae-4e6d-9737-6aef893fda01-kube-api-access-qtjn4\") pod \"maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962\" (UID: \"3a0b9f7e-5cae-4e6d-9737-6aef893fda01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962" Oct 30 17:48:40.322350 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:40.322328 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a0b9f7e-5cae-4e6d-9737-6aef893fda01-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962\" (UID: \"3a0b9f7e-5cae-4e6d-9737-6aef893fda01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962" Oct 30 17:48:40.322446 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:40.322431 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a0b9f7e-5cae-4e6d-9737-6aef893fda01-maas-api-tls\") pod \"maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962\" (UID: \"3a0b9f7e-5cae-4e6d-9737-6aef893fda01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962" Oct 30 17:48:40.323039 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:40.323011 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a0b9f7e-5cae-4e6d-9737-6aef893fda01-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962\" (UID: \"3a0b9f7e-5cae-4e6d-9737-6aef893fda01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962" Oct 30 17:48:40.324770 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:40.324748 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a0b9f7e-5cae-4e6d-9737-6aef893fda01-metrics-certs\") pod \"maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962\" (UID: \"3a0b9f7e-5cae-4e6d-9737-6aef893fda01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962" Oct 30 17:48:40.324965 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:40.324942 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a0b9f7e-5cae-4e6d-9737-6aef893fda01-maas-api-tls\") pod \"maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962\" (UID: \"3a0b9f7e-5cae-4e6d-9737-6aef893fda01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962" Oct 30 17:48:40.345964 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:40.345932 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtjn4\" (UniqueName: \"kubernetes.io/projected/3a0b9f7e-5cae-4e6d-9737-6aef893fda01-kube-api-access-qtjn4\") pod \"maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962\" (UID: \"3a0b9f7e-5cae-4e6d-9737-6aef893fda01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962" Oct 30 17:48:40.480058 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:40.479797 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962" Oct 30 17:48:40.811413 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:48:40.811378 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a0b9f7e_5cae_4e6d_9737_6aef893fda01.slice/crio-1413d4236cc0674d582496db35b123519f6fedce02bda231a65f316b58dd3e4f WatchSource:0}: Error finding container 1413d4236cc0674d582496db35b123519f6fedce02bda231a65f316b58dd3e4f: Status 404 returned error can't find the container with id 1413d4236cc0674d582496db35b123519f6fedce02bda231a65f316b58dd3e4f Oct 30 17:48:40.811848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:40.811820 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962"] Oct 30 17:48:40.813010 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:40.812990 4965 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:48:41.507940 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:41.505939 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-143041-6bfddb8867-5tnjk" podUID="f25e38d6-2f5a-4b43-bf0e-630706a343a1" containerName="payload-pre-processing" containerID="cri-o://0390d81e4e710549ea650c97a17a17698796927c245560f83d3a2f1e7bd89dd0" gracePeriod=25 Oct 30 17:48:41.700145 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:41.700100 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962" event={"ID":"3a0b9f7e-5cae-4e6d-9737-6aef893fda01","Type":"ContainerStarted","Data":"040992c2496f62a774fb072d58125b021815a4db92023699f49739b4753d8fc2"} Oct 30 17:48:41.700145 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:41.700179 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962" event={"ID":"3a0b9f7e-5cae-4e6d-9737-6aef893fda01","Type":"ContainerStarted","Data":"1413d4236cc0674d582496db35b123519f6fedce02bda231a65f316b58dd3e4f"} Oct 30 17:48:41.700145 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:41.700207 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962" Oct 30 17:48:41.750478 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:41.750421 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962" podStartSLOduration=1.397564504 podStartE2EDuration="1.750406514s" totalImagesPullingTime="352.84201ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:48:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:48:41.749634809 +0000 UTC m=+2179.801734740" watchObservedRunningTime="2026-10-30 17:48:41.750406514 +0000 UTC m=+2179.802506491" Oct 30 17:48:45.285800 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:45.285760 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-143041-6bfddb8867-5tnjk" podUID="f25e38d6-2f5a-4b43-bf0e-630706a343a1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:48:47.714528 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:47.714501 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962" Oct 30 17:48:55.286149 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:55.286108 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-143041-6bfddb8867-5tnjk" podUID="f25e38d6-2f5a-4b43-bf0e-630706a343a1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:48:56.236812 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:56.236777 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm"] Oct 30 17:48:56.237019 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:48:56.236997 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" podUID="c7db6ef5-c274-4ca7-b20b-e2eb17592720" containerName="istio-proxy" containerID="cri-o://67de36f2cc0a263f94c43e8d5cae63bb64585f66bb0f66d71f1772d011ec2f30" gracePeriod=30 Oct 30 17:49:01.669923 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.669894 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:49:01.810501 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.810469 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istio-envoy\") pod \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " Oct 30 17:49:01.810622 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.810565 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istio-podinfo\") pod \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " Oct 30 17:49:01.810677 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.810640 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-credential-socket\") pod \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " Oct 30 17:49:01.810739 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.810708 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7db6ef5-c274-4ca7-b20b-e2eb17592720-kube-api-access-p9n98\" (UniqueName: \"kubernetes.io/projected/c7db6ef5-c274-4ca7-b20b-e2eb17592720-kube-api-access-p9n98\") pod \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " Oct 30 17:49:01.810903 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.810845 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istio-token\" (UniqueName: \"kubernetes.io/projected/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istio-token\") pod \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " Oct 30 17:49:01.811086 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.811062 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istiod-ca-cert\") pod \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " Oct 30 17:49:01.811182 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.811154 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-workload-certs\") pod \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " Oct 30 17:49:01.811249 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.811185 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-credential-socket" pod "c7db6ef5-c274-4ca7-b20b-e2eb17592720" (UID: "c7db6ef5-c274-4ca7-b20b-e2eb17592720"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:01.811309 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.811262 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-workload-socket\") pod \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " Oct 30 17:49:01.811367 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.811305 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istio-data\") pod \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\" (UID: \"c7db6ef5-c274-4ca7-b20b-e2eb17592720\") " Oct 30 17:49:01.811735 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.811528 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-workload-certs" pod "c7db6ef5-c274-4ca7-b20b-e2eb17592720" (UID: "c7db6ef5-c274-4ca7-b20b-e2eb17592720"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:01.811735 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.811537 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-workload-socket" pod "c7db6ef5-c274-4ca7-b20b-e2eb17592720" (UID: "c7db6ef5-c274-4ca7-b20b-e2eb17592720"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:01.811735 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.811616 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istio-data" pod "c7db6ef5-c274-4ca7-b20b-e2eb17592720" (UID: "c7db6ef5-c274-4ca7-b20b-e2eb17592720"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:01.811735 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.811707 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istiod-ca-cert" pod "c7db6ef5-c274-4ca7-b20b-e2eb17592720" (UID: "c7db6ef5-c274-4ca7-b20b-e2eb17592720"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:49:01.812393 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.812369 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:01.812393 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.812398 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:01.812575 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.812412 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:01.812575 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.812426 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:01.812575 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.812439 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:01.813185 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.813153 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istio-envoy" pod "c7db6ef5-c274-4ca7-b20b-e2eb17592720" (UID: "c7db6ef5-c274-4ca7-b20b-e2eb17592720"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:01.813583 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.813540 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7db6ef5-c274-4ca7-b20b-e2eb17592720-kube-api-access-p9n98" pod "c7db6ef5-c274-4ca7-b20b-e2eb17592720" (UID: "c7db6ef5-c274-4ca7-b20b-e2eb17592720"). InnerVolumeSpecName "kube-api-access-p9n98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:49:01.813687 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.813587 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istio-podinfo" pod "c7db6ef5-c274-4ca7-b20b-e2eb17592720" (UID: "c7db6ef5-c274-4ca7-b20b-e2eb17592720"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:49:01.815215 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.814402 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istio-token" pod "c7db6ef5-c274-4ca7-b20b-e2eb17592720" (UID: "c7db6ef5-c274-4ca7-b20b-e2eb17592720"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:49:01.825368 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.825347 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-143041-6bfddb8867-5tnjk" Oct 30 17:49:01.913805 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.913764 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f25e38d6-2f5a-4b43-bf0e-630706a343a1-kube-api-access-59kpx\" (UniqueName: \"kubernetes.io/projected/f25e38d6-2f5a-4b43-bf0e-630706a343a1-kube-api-access-59kpx\") pod \"f25e38d6-2f5a-4b43-bf0e-630706a343a1\" (UID: \"f25e38d6-2f5a-4b43-bf0e-630706a343a1\") " Oct 30 17:49:01.914013 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.913914 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f25e38d6-2f5a-4b43-bf0e-630706a343a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f25e38d6-2f5a-4b43-bf0e-630706a343a1-plugins-config-volume\") pod \"f25e38d6-2f5a-4b43-bf0e-630706a343a1\" (UID: \"f25e38d6-2f5a-4b43-bf0e-630706a343a1\") " Oct 30 17:49:01.914276 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.914244 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f25e38d6-2f5a-4b43-bf0e-630706a343a1-plugins-config-volume" pod "f25e38d6-2f5a-4b43-bf0e-630706a343a1" (UID: "f25e38d6-2f5a-4b43-bf0e-630706a343a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:49:01.915019 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.914997 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:01.915080 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.915025 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:01.915080 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.915043 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7db6ef5-c274-4ca7-b20b-e2eb17592720-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:01.915080 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.915058 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f25e38d6-2f5a-4b43-bf0e-630706a343a1-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:01.915080 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.915074 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9n98\" (UniqueName: \"kubernetes.io/projected/c7db6ef5-c274-4ca7-b20b-e2eb17592720-kube-api-access-p9n98\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:01.916189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.916167 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f25e38d6-2f5a-4b43-bf0e-630706a343a1-kube-api-access-59kpx" pod "f25e38d6-2f5a-4b43-bf0e-630706a343a1" (UID: "f25e38d6-2f5a-4b43-bf0e-630706a343a1"). InnerVolumeSpecName "kube-api-access-59kpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:49:01.987943 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.987888 4965 generic.go:356] "Generic (PLEG): container finished" podID="c7db6ef5-c274-4ca7-b20b-e2eb17592720" containerID="67de36f2cc0a263f94c43e8d5cae63bb64585f66bb0f66d71f1772d011ec2f30" exitCode=0 Oct 30 17:49:01.988137 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.987967 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" Oct 30 17:49:01.988137 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.988000 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" event={"ID":"c7db6ef5-c274-4ca7-b20b-e2eb17592720","Type":"ContainerDied","Data":"67de36f2cc0a263f94c43e8d5cae63bb64585f66bb0f66d71f1772d011ec2f30"} Oct 30 17:49:01.988137 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.988060 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm" event={"ID":"c7db6ef5-c274-4ca7-b20b-e2eb17592720","Type":"ContainerDied","Data":"f189795347da6d93b4963c62ff76189cc2551d990416aad001f498edd0f59071"} Oct 30 17:49:01.988137 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.988082 4965 scope.go:122] "RemoveContainer" containerID="67de36f2cc0a263f94c43e8d5cae63bb64585f66bb0f66d71f1772d011ec2f30" Oct 30 17:49:01.989968 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.989940 4965 generic.go:356] "Generic (PLEG): container finished" podID="f25e38d6-2f5a-4b43-bf0e-630706a343a1" containerID="0390d81e4e710549ea650c97a17a17698796927c245560f83d3a2f1e7bd89dd0" exitCode=0 Oct 30 17:49:01.990104 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.990086 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-143041-6bfddb8867-5tnjk" event={"ID":"f25e38d6-2f5a-4b43-bf0e-630706a343a1","Type":"ContainerDied","Data":"0390d81e4e710549ea650c97a17a17698796927c245560f83d3a2f1e7bd89dd0"} Oct 30 17:49:01.990174 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.990134 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-143041-6bfddb8867-5tnjk" event={"ID":"f25e38d6-2f5a-4b43-bf0e-630706a343a1","Type":"ContainerDied","Data":"b4d82cad31336923e21e6f7a75ec77e542b8a880ca04dc5f31c6a748b873fcde"} Oct 30 17:49:01.990240 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:01.990222 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-143041-6bfddb8867-5tnjk" Oct 30 17:49:02.003185 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:02.003162 4965 scope.go:122] "RemoveContainer" containerID="67de36f2cc0a263f94c43e8d5cae63bb64585f66bb0f66d71f1772d011ec2f30" Oct 30 17:49:02.004532 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:49:02.004508 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67de36f2cc0a263f94c43e8d5cae63bb64585f66bb0f66d71f1772d011ec2f30\": container with ID starting with 67de36f2cc0a263f94c43e8d5cae63bb64585f66bb0f66d71f1772d011ec2f30 not found: ID does not exist" containerID="67de36f2cc0a263f94c43e8d5cae63bb64585f66bb0f66d71f1772d011ec2f30" Oct 30 17:49:02.004622 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:02.004546 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67de36f2cc0a263f94c43e8d5cae63bb64585f66bb0f66d71f1772d011ec2f30"} err="failed to get container status \"67de36f2cc0a263f94c43e8d5cae63bb64585f66bb0f66d71f1772d011ec2f30\": rpc error: code = NotFound desc = could not find container \"67de36f2cc0a263f94c43e8d5cae63bb64585f66bb0f66d71f1772d011ec2f30\": container with ID starting with 67de36f2cc0a263f94c43e8d5cae63bb64585f66bb0f66d71f1772d011ec2f30 not found: ID does not exist" Oct 30 17:49:02.004622 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:02.004571 4965 scope.go:122] "RemoveContainer" containerID="0390d81e4e710549ea650c97a17a17698796927c245560f83d3a2f1e7bd89dd0" Oct 30 17:49:02.017295 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:02.017268 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59kpx\" (UniqueName: \"kubernetes.io/projected/f25e38d6-2f5a-4b43-bf0e-630706a343a1-kube-api-access-59kpx\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:02.020505 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:02.020483 4965 scope.go:122] "RemoveContainer" containerID="0390d81e4e710549ea650c97a17a17698796927c245560f83d3a2f1e7bd89dd0" Oct 30 17:49:02.021036 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:49:02.021011 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0390d81e4e710549ea650c97a17a17698796927c245560f83d3a2f1e7bd89dd0\": container with ID starting with 0390d81e4e710549ea650c97a17a17698796927c245560f83d3a2f1e7bd89dd0 not found: ID does not exist" containerID="0390d81e4e710549ea650c97a17a17698796927c245560f83d3a2f1e7bd89dd0" Oct 30 17:49:02.021132 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:02.021043 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0390d81e4e710549ea650c97a17a17698796927c245560f83d3a2f1e7bd89dd0"} err="failed to get container status \"0390d81e4e710549ea650c97a17a17698796927c245560f83d3a2f1e7bd89dd0\": rpc error: code = NotFound desc = could not find container \"0390d81e4e710549ea650c97a17a17698796927c245560f83d3a2f1e7bd89dd0\": container with ID starting with 0390d81e4e710549ea650c97a17a17698796927c245560f83d3a2f1e7bd89dd0 not found: ID does not exist" Oct 30 17:49:02.044554 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:02.044518 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm"] Oct 30 17:49:02.048900 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:02.048821 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-8wptm"] Oct 30 17:49:02.085527 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:02.085494 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-143041-6bfddb8867-5tnjk"] Oct 30 17:49:02.089747 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:02.089720 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-143041-6bfddb8867-5tnjk"] Oct 30 17:49:02.101292 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:02.101262 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7db6ef5-c274-4ca7-b20b-e2eb17592720" path="/var/lib/kubelet/pods/c7db6ef5-c274-4ca7-b20b-e2eb17592720/volumes" Oct 30 17:49:02.101785 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:02.101770 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f25e38d6-2f5a-4b43-bf0e-630706a343a1" path="/var/lib/kubelet/pods/f25e38d6-2f5a-4b43-bf0e-630706a343a1/volumes" Oct 30 17:49:05.998730 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:49:05.998645 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:49:08.025016 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:08.024982 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-143041-5cf495455f-fcxtn"] Oct 30 17:49:08.025930 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:08.025174 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-143041-5cf495455f-fcxtn" podUID="51bd6cbd-5631-41f9-9dd8-be43832c6d32" containerName="maas-api" containerID="cri-o://7797d9af74c45352f287c4cabf796d04b6cf0ac11896d384759faae8014991f3" gracePeriod=30 Oct 30 17:49:08.732924 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:08.720519 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-143041-5cf495455f-fcxtn" Oct 30 17:49:08.811390 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:08.811357 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51bd6cbd-5631-41f9-9dd8-be43832c6d32-kube-api-access-nxdzk\" (UniqueName: \"kubernetes.io/projected/51bd6cbd-5631-41f9-9dd8-be43832c6d32-kube-api-access-nxdzk\") pod \"51bd6cbd-5631-41f9-9dd8-be43832c6d32\" (UID: \"51bd6cbd-5631-41f9-9dd8-be43832c6d32\") " Oct 30 17:49:08.811562 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:08.811438 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51bd6cbd-5631-41f9-9dd8-be43832c6d32-metrics-certs\" (UniqueName: \"kubernetes.io/secret/51bd6cbd-5631-41f9-9dd8-be43832c6d32-metrics-certs\") pod \"51bd6cbd-5631-41f9-9dd8-be43832c6d32\" (UID: \"51bd6cbd-5631-41f9-9dd8-be43832c6d32\") " Oct 30 17:49:08.811562 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:08.811492 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51bd6cbd-5631-41f9-9dd8-be43832c6d32-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51bd6cbd-5631-41f9-9dd8-be43832c6d32-maas-api-tls\") pod \"51bd6cbd-5631-41f9-9dd8-be43832c6d32\" (UID: \"51bd6cbd-5631-41f9-9dd8-be43832c6d32\") " Oct 30 17:49:08.811799 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:08.811774 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51bd6cbd-5631-41f9-9dd8-be43832c6d32-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51bd6cbd-5631-41f9-9dd8-be43832c6d32-openshift-service-ca\") pod \"51bd6cbd-5631-41f9-9dd8-be43832c6d32\" (UID: \"51bd6cbd-5631-41f9-9dd8-be43832c6d32\") " Oct 30 17:49:08.812353 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:08.812313 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51bd6cbd-5631-41f9-9dd8-be43832c6d32-openshift-service-ca" pod "51bd6cbd-5631-41f9-9dd8-be43832c6d32" (UID: "51bd6cbd-5631-41f9-9dd8-be43832c6d32"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:49:08.812749 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:08.812722 4965 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51bd6cbd-5631-41f9-9dd8-be43832c6d32-openshift-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:08.813992 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:08.813969 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51bd6cbd-5631-41f9-9dd8-be43832c6d32-kube-api-access-nxdzk" pod "51bd6cbd-5631-41f9-9dd8-be43832c6d32" (UID: "51bd6cbd-5631-41f9-9dd8-be43832c6d32"). InnerVolumeSpecName "kube-api-access-nxdzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:49:08.814109 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:08.814000 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51bd6cbd-5631-41f9-9dd8-be43832c6d32-metrics-certs" pod "51bd6cbd-5631-41f9-9dd8-be43832c6d32" (UID: "51bd6cbd-5631-41f9-9dd8-be43832c6d32"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:49:08.814179 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:08.814151 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51bd6cbd-5631-41f9-9dd8-be43832c6d32-maas-api-tls" pod "51bd6cbd-5631-41f9-9dd8-be43832c6d32" (UID: "51bd6cbd-5631-41f9-9dd8-be43832c6d32"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:49:08.914386 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:08.914292 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxdzk\" (UniqueName: \"kubernetes.io/projected/51bd6cbd-5631-41f9-9dd8-be43832c6d32-kube-api-access-nxdzk\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:08.914386 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:08.914326 4965 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51bd6cbd-5631-41f9-9dd8-be43832c6d32-metrics-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:08.914386 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:08.914336 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51bd6cbd-5631-41f9-9dd8-be43832c6d32-maas-api-tls\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:09.153259 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:09.146070 4965 generic.go:356] "Generic (PLEG): container finished" podID="51bd6cbd-5631-41f9-9dd8-be43832c6d32" containerID="7797d9af74c45352f287c4cabf796d04b6cf0ac11896d384759faae8014991f3" exitCode=0 Oct 30 17:49:09.153259 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:09.146324 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-143041-5cf495455f-fcxtn" event={"ID":"51bd6cbd-5631-41f9-9dd8-be43832c6d32","Type":"ContainerDied","Data":"7797d9af74c45352f287c4cabf796d04b6cf0ac11896d384759faae8014991f3"} Oct 30 17:49:09.153259 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:09.146377 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-143041-5cf495455f-fcxtn" event={"ID":"51bd6cbd-5631-41f9-9dd8-be43832c6d32","Type":"ContainerDied","Data":"4bd99483cda9e5c8cc4da62e9005044b202477f8478603f2811e712d680522bb"} Oct 30 17:49:09.153259 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:09.146406 4965 scope.go:122] "RemoveContainer" containerID="7797d9af74c45352f287c4cabf796d04b6cf0ac11896d384759faae8014991f3" Oct 30 17:49:09.153259 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:09.146593 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-143041-5cf495455f-fcxtn" Oct 30 17:49:09.167587 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:09.167563 4965 scope.go:122] "RemoveContainer" containerID="7797d9af74c45352f287c4cabf796d04b6cf0ac11896d384759faae8014991f3" Oct 30 17:49:09.167958 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:49:09.167896 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7797d9af74c45352f287c4cabf796d04b6cf0ac11896d384759faae8014991f3\": container with ID starting with 7797d9af74c45352f287c4cabf796d04b6cf0ac11896d384759faae8014991f3 not found: ID does not exist" containerID="7797d9af74c45352f287c4cabf796d04b6cf0ac11896d384759faae8014991f3" Oct 30 17:49:09.167958 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:09.167932 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7797d9af74c45352f287c4cabf796d04b6cf0ac11896d384759faae8014991f3"} err="failed to get container status \"7797d9af74c45352f287c4cabf796d04b6cf0ac11896d384759faae8014991f3\": rpc error: code = NotFound desc = could not find container \"7797d9af74c45352f287c4cabf796d04b6cf0ac11896d384759faae8014991f3\": container with ID starting with 7797d9af74c45352f287c4cabf796d04b6cf0ac11896d384759faae8014991f3 not found: ID does not exist" Oct 30 17:49:09.221921 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:09.221888 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-143041-5cf495455f-fcxtn"] Oct 30 17:49:09.229614 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:09.229585 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-143041-5cf495455f-fcxtn"] Oct 30 17:49:10.100883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:10.100831 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51bd6cbd-5631-41f9-9dd8-be43832c6d32" path="/var/lib/kubelet/pods/51bd6cbd-5631-41f9-9dd8-be43832c6d32/volumes" Oct 30 17:49:11.504721 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:11.504685 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt"] Oct 30 17:49:11.505322 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:11.504979 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" podUID="16d015c9-b3d7-45ae-a4f7-04aa68f678fd" containerName="istio-proxy" containerID="cri-o://ff301766bd8d874f52b0a792f0b64d7d642688e6dbf7b4acd1d777d72fdff97e" gracePeriod=30 Oct 30 17:49:11.837122 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:11.837088 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7"] Oct 30 17:49:11.837401 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:11.837345 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" podUID="9c3202ad-345e-4bb9-8987-c287fdef26a6" containerName="istio-proxy" containerID="cri-o://a6d4c776a2d0b8587e28290d0860da6c84d7754a1ab82136e38ce598da91ade9" gracePeriod=30 Oct 30 17:49:16.840977 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:16.840773 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:49:16.895822 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:16.895787 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istio-podinfo\") pod \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " Oct 30 17:49:16.896059 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:16.895844 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-workload-socket\") pod \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " Oct 30 17:49:16.896059 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:16.895880 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-credential-socket\") pod \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " Oct 30 17:49:16.896059 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:16.895920 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istio-envoy\") pod \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " Oct 30 17:49:16.896059 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:16.895991 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-workload-certs\") pod \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " Oct 30 17:49:16.896316 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:16.896127 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-workload-socket" pod "16d015c9-b3d7-45ae-a4f7-04aa68f678fd" (UID: "16d015c9-b3d7-45ae-a4f7-04aa68f678fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:16.896316 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:16.896142 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istio-data\") pod \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " Oct 30 17:49:16.896316 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:16.896235 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istiod-ca-cert\") pod \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " Oct 30 17:49:16.896446 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:16.896346 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istio-data" pod "16d015c9-b3d7-45ae-a4f7-04aa68f678fd" (UID: "16d015c9-b3d7-45ae-a4f7-04aa68f678fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:16.896446 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:16.896407 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istio-token\" (UniqueName: \"kubernetes.io/projected/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istio-token\") pod \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " Oct 30 17:49:16.896551 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:16.896526 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-kube-api-access-v82mz\" (UniqueName: \"kubernetes.io/projected/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-kube-api-access-v82mz\") pod \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\" (UID: \"16d015c9-b3d7-45ae-a4f7-04aa68f678fd\") " Oct 30 17:49:16.896657 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:16.896528 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-workload-certs" pod "16d015c9-b3d7-45ae-a4f7-04aa68f678fd" (UID: "16d015c9-b3d7-45ae-a4f7-04aa68f678fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:16.896720 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:16.896543 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-credential-socket" pod "16d015c9-b3d7-45ae-a4f7-04aa68f678fd" (UID: "16d015c9-b3d7-45ae-a4f7-04aa68f678fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:16.896776 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:16.896740 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istiod-ca-cert" pod "16d015c9-b3d7-45ae-a4f7-04aa68f678fd" (UID: "16d015c9-b3d7-45ae-a4f7-04aa68f678fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:49:16.897370 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:16.897340 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:16.897506 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:16.897375 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:16.897506 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:16.897390 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:16.897506 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:16.897404 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:16.897506 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:16.897417 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:16.898821 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:16.898795 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istio-envoy" pod "16d015c9-b3d7-45ae-a4f7-04aa68f678fd" (UID: "16d015c9-b3d7-45ae-a4f7-04aa68f678fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:16.899228 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:16.899205 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istio-podinfo" pod "16d015c9-b3d7-45ae-a4f7-04aa68f678fd" (UID: "16d015c9-b3d7-45ae-a4f7-04aa68f678fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:49:16.899415 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:16.899388 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-kube-api-access-v82mz" pod "16d015c9-b3d7-45ae-a4f7-04aa68f678fd" (UID: "16d015c9-b3d7-45ae-a4f7-04aa68f678fd"). InnerVolumeSpecName "kube-api-access-v82mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:49:16.899466 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:16.899447 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istio-token" pod "16d015c9-b3d7-45ae-a4f7-04aa68f678fd" (UID: "16d015c9-b3d7-45ae-a4f7-04aa68f678fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:49:16.999731 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:16.999643 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:16.999731 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:16.999672 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:16.999731 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:16.999686 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:17.000055 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:16.999698 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v82mz\" (UniqueName: \"kubernetes.io/projected/16d015c9-b3d7-45ae-a4f7-04aa68f678fd-kube-api-access-v82mz\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:17.227426 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.223218 4965 generic.go:356] "Generic (PLEG): container finished" podID="16d015c9-b3d7-45ae-a4f7-04aa68f678fd" containerID="ff301766bd8d874f52b0a792f0b64d7d642688e6dbf7b4acd1d777d72fdff97e" exitCode=0 Oct 30 17:49:17.227426 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.223291 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" Oct 30 17:49:17.227426 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.223522 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" event={"ID":"16d015c9-b3d7-45ae-a4f7-04aa68f678fd","Type":"ContainerDied","Data":"ff301766bd8d874f52b0a792f0b64d7d642688e6dbf7b4acd1d777d72fdff97e"} Oct 30 17:49:17.227426 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.223575 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt" event={"ID":"16d015c9-b3d7-45ae-a4f7-04aa68f678fd","Type":"ContainerDied","Data":"0fb408d1a34b0140bd5497dc6ecdc1833637751b00c421e6a6c18c5354bed4dc"} Oct 30 17:49:17.227426 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.223607 4965 scope.go:122] "RemoveContainer" containerID="ff301766bd8d874f52b0a792f0b64d7d642688e6dbf7b4acd1d777d72fdff97e" Oct 30 17:49:17.227426 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.226625 4965 generic.go:356] "Generic (PLEG): container finished" podID="9c3202ad-345e-4bb9-8987-c287fdef26a6" containerID="a6d4c776a2d0b8587e28290d0860da6c84d7754a1ab82136e38ce598da91ade9" exitCode=0 Oct 30 17:49:17.227426 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.226677 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" event={"ID":"9c3202ad-345e-4bb9-8987-c287fdef26a6","Type":"ContainerDied","Data":"a6d4c776a2d0b8587e28290d0860da6c84d7754a1ab82136e38ce598da91ade9"} Oct 30 17:49:17.241811 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.241768 4965 scope.go:122] "RemoveContainer" containerID="3756862eda02255279216d61f3e436bd949ab882cb498a8403ebde16a3168476" Oct 30 17:49:17.246297 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.246276 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:49:17.253804 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.253784 4965 scope.go:122] "RemoveContainer" containerID="ff301766bd8d874f52b0a792f0b64d7d642688e6dbf7b4acd1d777d72fdff97e" Oct 30 17:49:17.254124 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:49:17.254107 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff301766bd8d874f52b0a792f0b64d7d642688e6dbf7b4acd1d777d72fdff97e\": container with ID starting with ff301766bd8d874f52b0a792f0b64d7d642688e6dbf7b4acd1d777d72fdff97e not found: ID does not exist" containerID="ff301766bd8d874f52b0a792f0b64d7d642688e6dbf7b4acd1d777d72fdff97e" Oct 30 17:49:17.254183 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.254135 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff301766bd8d874f52b0a792f0b64d7d642688e6dbf7b4acd1d777d72fdff97e"} err="failed to get container status \"ff301766bd8d874f52b0a792f0b64d7d642688e6dbf7b4acd1d777d72fdff97e\": rpc error: code = NotFound desc = could not find container \"ff301766bd8d874f52b0a792f0b64d7d642688e6dbf7b4acd1d777d72fdff97e\": container with ID starting with ff301766bd8d874f52b0a792f0b64d7d642688e6dbf7b4acd1d777d72fdff97e not found: ID does not exist" Oct 30 17:49:17.254183 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.254153 4965 scope.go:122] "RemoveContainer" containerID="3756862eda02255279216d61f3e436bd949ab882cb498a8403ebde16a3168476" Oct 30 17:49:17.254444 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:49:17.254420 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3756862eda02255279216d61f3e436bd949ab882cb498a8403ebde16a3168476\": container with ID starting with 3756862eda02255279216d61f3e436bd949ab882cb498a8403ebde16a3168476 not found: ID does not exist" containerID="3756862eda02255279216d61f3e436bd949ab882cb498a8403ebde16a3168476" Oct 30 17:49:17.254540 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.254450 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3756862eda02255279216d61f3e436bd949ab882cb498a8403ebde16a3168476"} err="failed to get container status \"3756862eda02255279216d61f3e436bd949ab882cb498a8403ebde16a3168476\": rpc error: code = NotFound desc = could not find container \"3756862eda02255279216d61f3e436bd949ab882cb498a8403ebde16a3168476\": container with ID starting with 3756862eda02255279216d61f3e436bd949ab882cb498a8403ebde16a3168476 not found: ID does not exist" Oct 30 17:49:17.299633 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.299595 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt"] Oct 30 17:49:17.304381 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.304353 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-credential-socket\") pod \"9c3202ad-345e-4bb9-8987-c287fdef26a6\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " Oct 30 17:49:17.304532 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.304423 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c3202ad-345e-4bb9-8987-c287fdef26a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c3202ad-345e-4bb9-8987-c287fdef26a6-istiod-ca-cert\") pod \"9c3202ad-345e-4bb9-8987-c287fdef26a6\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " Oct 30 17:49:17.304532 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.304454 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c3202ad-345e-4bb9-8987-c287fdef26a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c3202ad-345e-4bb9-8987-c287fdef26a6-istio-podinfo\") pod \"9c3202ad-345e-4bb9-8987-c287fdef26a6\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " Oct 30 17:49:17.304532 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.304491 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-istio-data\") pod \"9c3202ad-345e-4bb9-8987-c287fdef26a6\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " Oct 30 17:49:17.304532 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.304518 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-workload-socket\") pod \"9c3202ad-345e-4bb9-8987-c287fdef26a6\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " Oct 30 17:49:17.304677 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.304647 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-credential-socket" pod "9c3202ad-345e-4bb9-8987-c287fdef26a6" (UID: "9c3202ad-345e-4bb9-8987-c287fdef26a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:17.304714 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.304695 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-workload-certs\") pod \"9c3202ad-345e-4bb9-8987-c287fdef26a6\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " Oct 30 17:49:17.304786 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.304769 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c3202ad-345e-4bb9-8987-c287fdef26a6-kube-api-access-vtwlh\" (UniqueName: \"kubernetes.io/projected/9c3202ad-345e-4bb9-8987-c287fdef26a6-kube-api-access-vtwlh\") pod \"9c3202ad-345e-4bb9-8987-c287fdef26a6\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " Oct 30 17:49:17.304830 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.304807 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-istio-envoy\") pod \"9c3202ad-345e-4bb9-8987-c287fdef26a6\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " Oct 30 17:49:17.304948 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.304923 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-workload-certs" pod "9c3202ad-345e-4bb9-8987-c287fdef26a6" (UID: "9c3202ad-345e-4bb9-8987-c287fdef26a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:17.305059 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.304953 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-workload-socket" pod "9c3202ad-345e-4bb9-8987-c287fdef26a6" (UID: "9c3202ad-345e-4bb9-8987-c287fdef26a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:17.305059 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.304978 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-istio-data" pod "9c3202ad-345e-4bb9-8987-c287fdef26a6" (UID: "9c3202ad-345e-4bb9-8987-c287fdef26a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:17.305059 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.304988 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c3202ad-345e-4bb9-8987-c287fdef26a6-istio-token\" (UniqueName: \"kubernetes.io/projected/9c3202ad-345e-4bb9-8987-c287fdef26a6-istio-token\") pod \"9c3202ad-345e-4bb9-8987-c287fdef26a6\" (UID: \"9c3202ad-345e-4bb9-8987-c287fdef26a6\") " Oct 30 17:49:17.305059 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.305015 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c3202ad-345e-4bb9-8987-c287fdef26a6-istiod-ca-cert" pod "9c3202ad-345e-4bb9-8987-c287fdef26a6" (UID: "9c3202ad-345e-4bb9-8987-c287fdef26a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:49:17.305730 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.305708 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:17.305730 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.305731 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c3202ad-345e-4bb9-8987-c287fdef26a6-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:17.305842 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.305741 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:17.305842 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.305749 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:17.305842 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.305757 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:17.307087 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.307066 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c3202ad-345e-4bb9-8987-c287fdef26a6-istio-podinfo" pod "9c3202ad-345e-4bb9-8987-c287fdef26a6" (UID: "9c3202ad-345e-4bb9-8987-c287fdef26a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:49:17.307408 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.307390 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c3202ad-345e-4bb9-8987-c287fdef26a6-istio-token" pod "9c3202ad-345e-4bb9-8987-c287fdef26a6" (UID: "9c3202ad-345e-4bb9-8987-c287fdef26a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:49:17.307523 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.307503 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c3202ad-345e-4bb9-8987-c287fdef26a6-kube-api-access-vtwlh" pod "9c3202ad-345e-4bb9-8987-c287fdef26a6" (UID: "9c3202ad-345e-4bb9-8987-c287fdef26a6"). InnerVolumeSpecName "kube-api-access-vtwlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:49:17.307583 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.307558 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-istio-envoy" pod "9c3202ad-345e-4bb9-8987-c287fdef26a6" (UID: "9c3202ad-345e-4bb9-8987-c287fdef26a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:17.311057 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.311036 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-vsttt"] Oct 30 17:49:17.408951 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.408250 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtwlh\" (UniqueName: \"kubernetes.io/projected/9c3202ad-345e-4bb9-8987-c287fdef26a6-kube-api-access-vtwlh\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:17.408951 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.408305 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c3202ad-345e-4bb9-8987-c287fdef26a6-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:17.408951 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.408319 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c3202ad-345e-4bb9-8987-c287fdef26a6-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:17.408951 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.408333 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c3202ad-345e-4bb9-8987-c287fdef26a6-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:17.780480 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.770578 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn"] Oct 30 17:49:17.780480 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.772627 4965 cpu_manager.go:414] "containerMap: removing container" podUID="c7db6ef5-c274-4ca7-b20b-e2eb17592720" containerName="istio-proxy" Oct 30 17:49:17.780480 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.772651 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7db6ef5-c274-4ca7-b20b-e2eb17592720" containerName="istio-proxy" Oct 30 17:49:17.780480 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.772682 4965 cpu_manager.go:414] "containerMap: removing container" podUID="f25e38d6-2f5a-4b43-bf0e-630706a343a1" containerName="payload-pre-processing" Oct 30 17:49:17.780480 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.772690 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f25e38d6-2f5a-4b43-bf0e-630706a343a1" containerName="payload-pre-processing" Oct 30 17:49:17.780480 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.772766 4965 cpu_manager.go:414] "containerMap: removing container" podUID="51bd6cbd-5631-41f9-9dd8-be43832c6d32" containerName="maas-api" Oct 30 17:49:17.780480 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.772775 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51bd6cbd-5631-41f9-9dd8-be43832c6d32" containerName="maas-api" Oct 30 17:49:17.780480 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.772848 4965 cpu_manager.go:414] "containerMap: removing container" podUID="16d015c9-b3d7-45ae-a4f7-04aa68f678fd" containerName="istio-proxy" Oct 30 17:49:17.914937 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.772855 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16d015c9-b3d7-45ae-a4f7-04aa68f678fd" containerName="istio-proxy" Oct 30 17:49:17.914937 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.882669 4965 cpu_manager.go:414] "containerMap: removing container" podUID="16d015c9-b3d7-45ae-a4f7-04aa68f678fd" containerName="istio-proxy" Oct 30 17:49:17.914937 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.882693 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16d015c9-b3d7-45ae-a4f7-04aa68f678fd" containerName="istio-proxy" Oct 30 17:49:17.914937 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.882747 4965 cpu_manager.go:414] "containerMap: removing container" podUID="9c3202ad-345e-4bb9-8987-c287fdef26a6" containerName="istio-proxy" Oct 30 17:49:17.914937 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.882755 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c3202ad-345e-4bb9-8987-c287fdef26a6" containerName="istio-proxy" Oct 30 17:49:17.914937 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.883467 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="16d015c9-b3d7-45ae-a4f7-04aa68f678fd" containerName="istio-proxy" Oct 30 17:49:17.914937 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.883499 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="16d015c9-b3d7-45ae-a4f7-04aa68f678fd" containerName="istio-proxy" Oct 30 17:49:17.914937 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.883524 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c3202ad-345e-4bb9-8987-c287fdef26a6" containerName="istio-proxy" Oct 30 17:49:17.914937 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.883544 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7db6ef5-c274-4ca7-b20b-e2eb17592720" containerName="istio-proxy" Oct 30 17:49:17.914937 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.883556 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="f25e38d6-2f5a-4b43-bf0e-630706a343a1" containerName="payload-pre-processing" Oct 30 17:49:17.914937 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.883610 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="51bd6cbd-5631-41f9-9dd8-be43832c6d32" containerName="maas-api" Oct 30 17:49:17.914937 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.883630 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a0c4b63-604c-49de-9f33-4465aec216bc" containerName="maas-api" Oct 30 17:49:18.017937 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:17.999089 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:18.045291 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.045242 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-workload-certs\") pod \"e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:18.045470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.045349 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-istio-data\") pod \"e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:18.045550 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.045524 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48e4e05-e95d-424c-887e-668bdc42e398-istiod-ca-cert\") pod \"e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:18.045683 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.045667 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-credential-socket\") pod \"e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:18.045739 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.045715 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48e4e05-e95d-424c-887e-668bdc42e398-istio-podinfo\") pod \"e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:18.045795 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.045777 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddgrr\" (UniqueName: \"kubernetes.io/projected/b48e4e05-e95d-424c-887e-668bdc42e398-kube-api-access-ddgrr\") pod \"e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:18.045952 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.045829 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b48e4e05-e95d-424c-887e-668bdc42e398-istio-token\") pod \"e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:18.046009 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.045957 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-istio-envoy\") pod \"e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:18.046072 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.046040 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-workload-socket\") pod \"e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:18.051377 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.051350 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn"] Oct 30 17:49:18.054037 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.054014 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-ce0480-openshift-default-dockercfg-8qgqf\"" Oct 30 17:49:18.105009 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.104664 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16d015c9-b3d7-45ae-a4f7-04aa68f678fd" path="/var/lib/kubelet/pods/16d015c9-b3d7-45ae-a4f7-04aa68f678fd/volumes" Oct 30 17:49:18.147818 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.147783 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-credential-socket\") pod \"e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:18.148070 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.147830 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48e4e05-e95d-424c-887e-668bdc42e398-istio-podinfo\") pod \"e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:18.148070 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.147899 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddgrr\" (UniqueName: \"kubernetes.io/projected/b48e4e05-e95d-424c-887e-668bdc42e398-kube-api-access-ddgrr\") pod \"e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:18.148070 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.147930 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b48e4e05-e95d-424c-887e-668bdc42e398-istio-token\") pod \"e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:18.148070 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.148003 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-istio-envoy\") pod \"e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:18.148070 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.148052 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-workload-socket\") pod \"e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:18.148366 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.148085 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-workload-certs\") pod \"e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:18.148366 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.148170 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-istio-data\") pod \"e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:18.148366 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.148298 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48e4e05-e95d-424c-887e-668bdc42e398-istiod-ca-cert\") pod \"e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:18.148550 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.148506 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-credential-socket\") pod \"e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:18.148935 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.148904 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-workload-certs\") pod \"e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:18.149045 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.148906 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-istio-data\") pod \"e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:18.149202 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.149178 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-workload-socket\") pod \"e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:18.149372 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.149273 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48e4e05-e95d-424c-887e-668bdc42e398-istiod-ca-cert\") pod \"e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:18.150963 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.150942 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-istio-envoy\") pod \"e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:18.151633 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.151612 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48e4e05-e95d-424c-887e-668bdc42e398-istio-podinfo\") pod \"e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:18.186607 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.186533 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddgrr\" (UniqueName: \"kubernetes.io/projected/b48e4e05-e95d-424c-887e-668bdc42e398-kube-api-access-ddgrr\") pod \"e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:18.187159 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.187133 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b48e4e05-e95d-424c-887e-668bdc42e398-istio-token\") pod \"e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:18.251173 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.250799 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" event={"ID":"9c3202ad-345e-4bb9-8987-c287fdef26a6","Type":"ContainerDied","Data":"b960a6b13533c956dc3dbc4dbf18919925c39b1526462d6c643d2a5417446593"} Oct 30 17:49:18.252147 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.251476 4965 scope.go:122] "RemoveContainer" containerID="a6d4c776a2d0b8587e28290d0860da6c84d7754a1ab82136e38ce598da91ade9" Oct 30 17:49:18.252147 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.251699 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7" Oct 30 17:49:18.334995 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.334967 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7"] Oct 30 17:49:18.339558 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.339530 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-h57d7"] Oct 30 17:49:18.378888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.378128 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:18.838108 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:49:18.838078 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb48e4e05_e95d_424c_887e_668bdc42e398.slice/crio-b6c2783031b1519795c2236a8ec21904c738fa1faf203deab5fdb65c909790a6 WatchSource:0}: Error finding container b6c2783031b1519795c2236a8ec21904c738fa1faf203deab5fdb65c909790a6: Status 404 returned error can't find the container with id b6c2783031b1519795c2236a8ec21904c738fa1faf203deab5fdb65c909790a6 Oct 30 17:49:18.838244 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.838135 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn"] Oct 30 17:49:18.840827 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.840792 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:49:18.840993 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.840898 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:49:18.840993 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:18.840935 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:49:19.295598 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:19.295491 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" event={"ID":"b48e4e05-e95d-424c-887e-668bdc42e398","Type":"ContainerStarted","Data":"c496ac3cc83d4845fddb5f3a7d2d11aba86a1c1693c78d96f222c97494166752"} Oct 30 17:49:19.295598 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:19.295569 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" event={"ID":"b48e4e05-e95d-424c-887e-668bdc42e398","Type":"ContainerStarted","Data":"b6c2783031b1519795c2236a8ec21904c738fa1faf203deab5fdb65c909790a6"} Oct 30 17:49:19.385885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:19.385459 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:19.387887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:19.387225 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 17:49:19.387887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:19.387301 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" podUID="b48e4e05-e95d-424c-887e-668bdc42e398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 17:49:20.101401 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:20.101367 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c3202ad-345e-4bb9-8987-c287fdef26a6" path="/var/lib/kubelet/pods/9c3202ad-345e-4bb9-8987-c287fdef26a6/volumes" Oct 30 17:49:20.390483 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:20.390411 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:20.453647 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:20.453599 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" podStartSLOduration=3.453584552 podStartE2EDuration="3.453584552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:49:19.344509604 +0000 UTC m=+2217.396609535" watchObservedRunningTime="2026-10-30 17:49:20.453584552 +0000 UTC m=+2218.505684564" Oct 30 17:49:21.311885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:21.311842 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:21.312925 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:21.312898 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:49:24.018047 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:24.017962 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ce0480-5cd9bdc47d-974w6"] Oct 30 17:49:24.081432 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:24.081399 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ce0480-5cd9bdc47d-974w6"] Oct 30 17:49:24.083763 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:24.081703 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ce0480-5cd9bdc47d-974w6" Oct 30 17:49:24.101695 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:24.101668 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-ce0480\"" Oct 30 17:49:24.115228 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:24.115203 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-ce0480-dockercfg-srldd\"" Oct 30 17:49:24.181614 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:24.181569 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2721707-097c-4adf-a415-15be655b9729-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-ce0480-5cd9bdc47d-974w6\" (UID: \"f2721707-097c-4adf-a415-15be655b9729\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ce0480-5cd9bdc47d-974w6" Oct 30 17:49:24.181973 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:24.181945 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt868\" (UniqueName: \"kubernetes.io/projected/f2721707-097c-4adf-a415-15be655b9729-kube-api-access-zt868\") pod \"payload-processing-e2e-shared-a-w1-ce0480-5cd9bdc47d-974w6\" (UID: \"f2721707-097c-4adf-a415-15be655b9729\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ce0480-5cd9bdc47d-974w6" Oct 30 17:49:24.284180 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:24.284095 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt868\" (UniqueName: \"kubernetes.io/projected/f2721707-097c-4adf-a415-15be655b9729-kube-api-access-zt868\") pod \"payload-processing-e2e-shared-a-w1-ce0480-5cd9bdc47d-974w6\" (UID: \"f2721707-097c-4adf-a415-15be655b9729\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ce0480-5cd9bdc47d-974w6" Oct 30 17:49:24.284345 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:24.284304 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2721707-097c-4adf-a415-15be655b9729-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-ce0480-5cd9bdc47d-974w6\" (UID: \"f2721707-097c-4adf-a415-15be655b9729\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ce0480-5cd9bdc47d-974w6" Oct 30 17:49:24.285051 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:24.285027 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2721707-097c-4adf-a415-15be655b9729-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-ce0480-5cd9bdc47d-974w6\" (UID: \"f2721707-097c-4adf-a415-15be655b9729\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ce0480-5cd9bdc47d-974w6" Oct 30 17:49:24.308079 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:24.308037 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt868\" (UniqueName: \"kubernetes.io/projected/f2721707-097c-4adf-a415-15be655b9729-kube-api-access-zt868\") pod \"payload-processing-e2e-shared-a-w1-ce0480-5cd9bdc47d-974w6\" (UID: \"f2721707-097c-4adf-a415-15be655b9729\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ce0480-5cd9bdc47d-974w6" Oct 30 17:49:24.420887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:24.411371 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ce0480-5cd9bdc47d-974w6" Oct 30 17:49:24.807769 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:24.807732 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ce0480-5cd9bdc47d-974w6"] Oct 30 17:49:24.807924 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:49:24.807728 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2721707_097c_4adf_a415_15be655b9729.slice/crio-08172d9e3001fdbcb59b03a3f02a408c6f41d774e143acf48500d20eaad8d1c6 WatchSource:0}: Error finding container 08172d9e3001fdbcb59b03a3f02a408c6f41d774e143acf48500d20eaad8d1c6: Status 404 returned error can't find the container with id 08172d9e3001fdbcb59b03a3f02a408c6f41d774e143acf48500d20eaad8d1c6 Oct 30 17:49:25.388573 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:25.387878 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ce0480-5cd9bdc47d-974w6" event={"ID":"f2721707-097c-4adf-a415-15be655b9729","Type":"ContainerStarted","Data":"a3fb17e3d2c78d0626a185335a275d8083e458574994cae161f955e27803170a"} Oct 30 17:49:25.388573 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:25.387941 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ce0480-5cd9bdc47d-974w6" event={"ID":"f2721707-097c-4adf-a415-15be655b9729","Type":"ContainerStarted","Data":"08172d9e3001fdbcb59b03a3f02a408c6f41d774e143acf48500d20eaad8d1c6"} Oct 30 17:49:25.388573 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:25.387972 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ce0480-5cd9bdc47d-974w6" Oct 30 17:49:25.429905 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:25.429831 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ce0480-5cd9bdc47d-974w6" podStartSLOduration=2.429818203 podStartE2EDuration="2.429818203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:49:25.42763027 +0000 UTC m=+2223.479730201" watchObservedRunningTime="2026-10-30 17:49:25.429818203 +0000 UTC m=+2223.481918133" Oct 30 17:49:27.405612 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:27.405582 4965 generic.go:356] "Generic (PLEG): container finished" podID="50d223e1-883f-454f-9c29-348129b7d676" containerID="0d0c0f6ce3e971f2cab20611549e4304e190a972dcf33697fd4ecc0fc7b165df" exitCode=0 Oct 30 17:49:27.406110 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:27.405650 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" event={"ID":"50d223e1-883f-454f-9c29-348129b7d676","Type":"ContainerDied","Data":"0d0c0f6ce3e971f2cab20611549e4304e190a972dcf33697fd4ecc0fc7b165df"} Oct 30 17:49:27.406198 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:27.406183 4965 scope.go:122] "RemoveContainer" containerID="0d0c0f6ce3e971f2cab20611549e4304e190a972dcf33697fd4ecc0fc7b165df" Oct 30 17:49:27.408301 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:27.408270 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:49:27.408394 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:27.408342 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:49:27.408394 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:27.408375 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:49:28.419260 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:28.419203 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" event={"ID":"50d223e1-883f-454f-9c29-348129b7d676","Type":"ContainerStarted","Data":"2ca97e7cfca2c2246fab3e6871885e92a10dffa92c8fae7a0e6576dec6495b0b"} Oct 30 17:49:28.741773 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:28.738641 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:49:28.755349 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:28.755310 4965 patch_prober.go:29] interesting pod/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 30 17:49:28.764008 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:28.763961 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" podUID="50d223e1-883f-454f-9c29-348129b7d676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 30 17:49:29.722887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:29.718051 4965 patch_prober.go:29] interesting pod/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 30 17:49:29.722887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:29.718124 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" podUID="50d223e1-883f-454f-9c29-348129b7d676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 30 17:49:30.534486 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:30.534449 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:49:30.695183 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:30.695149 4965 patch_prober.go:29] interesting pod/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 30 17:49:30.695455 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:30.695425 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" podUID="50d223e1-883f-454f-9c29-348129b7d676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 30 17:49:31.697953 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:31.697358 4965 patch_prober.go:29] interesting pod/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 30 17:49:31.697953 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:31.697488 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" podUID="50d223e1-883f-454f-9c29-348129b7d676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 30 17:49:32.711453 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:32.711413 4965 patch_prober.go:29] interesting pod/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 30 17:49:32.712285 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:32.712140 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" podUID="50d223e1-883f-454f-9c29-348129b7d676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 30 17:49:33.695937 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:33.695906 4965 patch_prober.go:29] interesting pod/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 30 17:49:33.696146 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:33.695957 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" podUID="50d223e1-883f-454f-9c29-348129b7d676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 30 17:49:34.783512 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:34.783473 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:49:34.784065 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:34.784045 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:49:34.784962 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:34.784938 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:49:35.998739 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:49:35.998704 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:49:36.401970 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:36.401940 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ce0480-5cd9bdc47d-974w6" Oct 30 17:49:49.240816 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:49.240780 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62"] Oct 30 17:49:49.241468 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:49.241012 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62" podUID="cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca" containerName="maas-api" containerID="cri-o://f1cebd30075e3e6bf733a83dafa413f1bf6128b2005b3cbb19a87a2e53c355fc" gracePeriod=30 Oct 30 17:49:49.863208 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:49.863170 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62" Oct 30 17:49:49.881522 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:49.881487 4965 generic.go:356] "Generic (PLEG): container finished" podID="cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca" containerID="f1cebd30075e3e6bf733a83dafa413f1bf6128b2005b3cbb19a87a2e53c355fc" exitCode=0 Oct 30 17:49:49.882060 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:49.882032 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62" event={"ID":"cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca","Type":"ContainerDied","Data":"f1cebd30075e3e6bf733a83dafa413f1bf6128b2005b3cbb19a87a2e53c355fc"} Oct 30 17:49:49.882459 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:49.882439 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62" event={"ID":"cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca","Type":"ContainerDied","Data":"feb3efef6cddd9cebbf0c9084d6a198c7fdce678853c160fcc523a7a06c6b6cc"} Oct 30 17:49:49.882615 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:49.882602 4965 scope.go:122] "RemoveContainer" containerID="f1cebd30075e3e6bf733a83dafa413f1bf6128b2005b3cbb19a87a2e53c355fc" Oct 30 17:49:49.882814 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:49.882382 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62" Oct 30 17:49:49.950271 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:49.950237 4965 scope.go:122] "RemoveContainer" containerID="f1cebd30075e3e6bf733a83dafa413f1bf6128b2005b3cbb19a87a2e53c355fc" Oct 30 17:49:49.952837 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:49:49.952758 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1cebd30075e3e6bf733a83dafa413f1bf6128b2005b3cbb19a87a2e53c355fc\": container with ID starting with f1cebd30075e3e6bf733a83dafa413f1bf6128b2005b3cbb19a87a2e53c355fc not found: ID does not exist" containerID="f1cebd30075e3e6bf733a83dafa413f1bf6128b2005b3cbb19a87a2e53c355fc" Oct 30 17:49:49.953083 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:49.953057 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1cebd30075e3e6bf733a83dafa413f1bf6128b2005b3cbb19a87a2e53c355fc"} err="failed to get container status \"f1cebd30075e3e6bf733a83dafa413f1bf6128b2005b3cbb19a87a2e53c355fc\": rpc error: code = NotFound desc = could not find container \"f1cebd30075e3e6bf733a83dafa413f1bf6128b2005b3cbb19a87a2e53c355fc\": container with ID starting with f1cebd30075e3e6bf733a83dafa413f1bf6128b2005b3cbb19a87a2e53c355fc not found: ID does not exist" Oct 30 17:49:49.984584 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:49.982342 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca-maas-api-tls\") pod \"cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca\" (UID: \"cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca\") " Oct 30 17:49:49.984584 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:49.982421 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca-metrics-certs\") pod \"cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca\" (UID: \"cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca\") " Oct 30 17:49:49.984584 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:49.982512 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca-openshift-service-ca\") pod \"cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca\" (UID: \"cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca\") " Oct 30 17:49:49.984584 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:49.982562 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca-kube-api-access-qsvlw\" (UniqueName: \"kubernetes.io/projected/cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca-kube-api-access-qsvlw\") pod \"cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca\" (UID: \"cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca\") " Oct 30 17:49:49.984584 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:49.984116 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca-openshift-service-ca" pod "cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca" (UID: "cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:49:49.986246 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:49.986215 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca-kube-api-access-qsvlw" pod "cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca" (UID: "cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca"). InnerVolumeSpecName "kube-api-access-qsvlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:49:49.986576 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:49.986555 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca-metrics-certs" pod "cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca" (UID: "cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:49:49.987754 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:49.987728 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca-maas-api-tls" pod "cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca" (UID: "cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:49:50.085323 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:50.085290 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca-maas-api-tls\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:50.085323 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:50.085319 4965 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca-metrics-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:50.085323 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:50.085328 4965 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca-openshift-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:50.085567 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:50.085338 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsvlw\" (UniqueName: \"kubernetes.io/projected/cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca-kube-api-access-qsvlw\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:49:50.225366 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:50.225283 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62"] Oct 30 17:49:50.236759 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:50.236729 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-874b83-6fb5df8479-6tc62"] Oct 30 17:49:52.119772 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:52.119729 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca" path="/var/lib/kubelet/pods/cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca/volumes" Oct 30 17:49:57.932134 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:57.932098 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh"] Oct 30 17:49:57.932947 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:57.932915 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" podUID="33457c12-09bd-4268-83a6-3543bd62be67" containerName="istio-proxy" containerID="cri-o://edb8ac61d2c3c719c9905e8c1632468fdad0efd0aa0daa0a05efb58ce85a3fd0" gracePeriod=30 Oct 30 17:49:57.941966 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:57.941939 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g"] Oct 30 17:49:57.969352 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:49:57.969319 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" podUID="508ddd74-da86-4674-ad80-7af4a1eb7310" containerName="istio-proxy" containerID="cri-o://f294af8eb72ab92728967dc99939fd78b0d055ab55dbe21e483a3559cdc0f587" gracePeriod=30 Oct 30 17:50:01.251964 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:01.251927 4965 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:50:01.252361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:01.251998 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" podUID="33457c12-09bd-4268-83a6-3543bd62be67" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:50:02.057674 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:02.057625 4965 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:50:02.057931 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:02.057694 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" podUID="508ddd74-da86-4674-ad80-7af4a1eb7310" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:50:03.015029 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.014999 4965 generic.go:356] "Generic (PLEG): container finished" podID="508ddd74-da86-4674-ad80-7af4a1eb7310" containerID="f294af8eb72ab92728967dc99939fd78b0d055ab55dbe21e483a3559cdc0f587" exitCode=0 Oct 30 17:50:03.015393 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.015279 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" event={"ID":"508ddd74-da86-4674-ad80-7af4a1eb7310","Type":"ContainerDied","Data":"f294af8eb72ab92728967dc99939fd78b0d055ab55dbe21e483a3559cdc0f587"} Oct 30 17:50:03.015393 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.015342 4965 scope.go:122] "RemoveContainer" containerID="a00bc22a52abafa1d7649f89563b915129c6562f936fc8ef33227ab8d4aff299" Oct 30 17:50:03.019234 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.019210 4965 generic.go:356] "Generic (PLEG): container finished" podID="33457c12-09bd-4268-83a6-3543bd62be67" containerID="edb8ac61d2c3c719c9905e8c1632468fdad0efd0aa0daa0a05efb58ce85a3fd0" exitCode=0 Oct 30 17:50:03.019382 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.019255 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" event={"ID":"33457c12-09bd-4268-83a6-3543bd62be67","Type":"ContainerDied","Data":"edb8ac61d2c3c719c9905e8c1632468fdad0efd0aa0daa0a05efb58ce85a3fd0"} Oct 30 17:50:03.317980 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.317946 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:50:03.369495 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.369467 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:50:03.414162 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.414122 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33457c12-09bd-4268-83a6-3543bd62be67-kube-api-access-gd9q9\" (UniqueName: \"kubernetes.io/projected/33457c12-09bd-4268-83a6-3543bd62be67-kube-api-access-gd9q9\") pod \"33457c12-09bd-4268-83a6-3543bd62be67\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " Oct 30 17:50:03.414377 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.414231 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33457c12-09bd-4268-83a6-3543bd62be67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33457c12-09bd-4268-83a6-3543bd62be67-istiod-ca-cert\") pod \"33457c12-09bd-4268-83a6-3543bd62be67\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " Oct 30 17:50:03.414377 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.414281 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/508ddd74-da86-4674-ad80-7af4a1eb7310-kube-api-access-xvzps\" (UniqueName: \"kubernetes.io/projected/508ddd74-da86-4674-ad80-7af4a1eb7310-kube-api-access-xvzps\") pod \"508ddd74-da86-4674-ad80-7af4a1eb7310\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " Oct 30 17:50:03.414377 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.414316 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-istio-data\") pod \"33457c12-09bd-4268-83a6-3543bd62be67\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " Oct 30 17:50:03.414377 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.414352 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33457c12-09bd-4268-83a6-3543bd62be67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33457c12-09bd-4268-83a6-3543bd62be67-istio-podinfo\") pod \"33457c12-09bd-4268-83a6-3543bd62be67\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " Oct 30 17:50:03.414625 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.414422 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-istio-envoy\") pod \"33457c12-09bd-4268-83a6-3543bd62be67\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " Oct 30 17:50:03.414625 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.414461 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-credential-socket\") pod \"508ddd74-da86-4674-ad80-7af4a1eb7310\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " Oct 30 17:50:03.414741 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.414638 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-workload-certs\") pod \"508ddd74-da86-4674-ad80-7af4a1eb7310\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " Oct 30 17:50:03.414741 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.414699 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/508ddd74-da86-4674-ad80-7af4a1eb7310-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/508ddd74-da86-4674-ad80-7af4a1eb7310-istio-podinfo\") pod \"508ddd74-da86-4674-ad80-7af4a1eb7310\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " Oct 30 17:50:03.414848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.414759 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-workload-certs\") pod \"33457c12-09bd-4268-83a6-3543bd62be67\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " Oct 30 17:50:03.414848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.414792 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-istio-data" pod "33457c12-09bd-4268-83a6-3543bd62be67" (UID: "33457c12-09bd-4268-83a6-3543bd62be67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:50:03.414848 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.414829 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/508ddd74-da86-4674-ad80-7af4a1eb7310-istio-token\" (UniqueName: \"kubernetes.io/projected/508ddd74-da86-4674-ad80-7af4a1eb7310-istio-token\") pod \"508ddd74-da86-4674-ad80-7af4a1eb7310\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " Oct 30 17:50:03.415033 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.414891 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-istio-envoy\") pod \"508ddd74-da86-4674-ad80-7af4a1eb7310\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " Oct 30 17:50:03.415033 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.414945 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-credential-socket\") pod \"33457c12-09bd-4268-83a6-3543bd62be67\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " Oct 30 17:50:03.415033 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.414992 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-workload-socket\") pod \"508ddd74-da86-4674-ad80-7af4a1eb7310\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " Oct 30 17:50:03.415179 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.415031 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-workload-socket\") pod \"33457c12-09bd-4268-83a6-3543bd62be67\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " Oct 30 17:50:03.415179 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.415073 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/508ddd74-da86-4674-ad80-7af4a1eb7310-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/508ddd74-da86-4674-ad80-7af4a1eb7310-istiod-ca-cert\") pod \"508ddd74-da86-4674-ad80-7af4a1eb7310\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " Oct 30 17:50:03.415179 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.415110 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-istio-data\") pod \"508ddd74-da86-4674-ad80-7af4a1eb7310\" (UID: \"508ddd74-da86-4674-ad80-7af4a1eb7310\") " Oct 30 17:50:03.418700 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.415192 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33457c12-09bd-4268-83a6-3543bd62be67-istiod-ca-cert" pod "33457c12-09bd-4268-83a6-3543bd62be67" (UID: "33457c12-09bd-4268-83a6-3543bd62be67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:50:03.418700 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.415256 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33457c12-09bd-4268-83a6-3543bd62be67-istio-token\" (UniqueName: \"kubernetes.io/projected/33457c12-09bd-4268-83a6-3543bd62be67-istio-token\") pod \"33457c12-09bd-4268-83a6-3543bd62be67\" (UID: \"33457c12-09bd-4268-83a6-3543bd62be67\") " Oct 30 17:50:03.418700 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.416108 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33457c12-09bd-4268-83a6-3543bd62be67-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:50:03.418700 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.416133 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:50:03.418700 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.416411 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-workload-socket" pod "508ddd74-da86-4674-ad80-7af4a1eb7310" (UID: "508ddd74-da86-4674-ad80-7af4a1eb7310"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:50:03.418700 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.416670 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-credential-socket" pod "508ddd74-da86-4674-ad80-7af4a1eb7310" (UID: "508ddd74-da86-4674-ad80-7af4a1eb7310"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:50:03.418700 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.417193 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-workload-certs" pod "508ddd74-da86-4674-ad80-7af4a1eb7310" (UID: "508ddd74-da86-4674-ad80-7af4a1eb7310"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:50:03.418700 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.417519 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/508ddd74-da86-4674-ad80-7af4a1eb7310-istiod-ca-cert" pod "508ddd74-da86-4674-ad80-7af4a1eb7310" (UID: "508ddd74-da86-4674-ad80-7af4a1eb7310"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:50:03.418700 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.417830 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33457c12-09bd-4268-83a6-3543bd62be67-kube-api-access-gd9q9" pod "33457c12-09bd-4268-83a6-3543bd62be67" (UID: "33457c12-09bd-4268-83a6-3543bd62be67"). InnerVolumeSpecName "kube-api-access-gd9q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:50:03.418700 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.418010 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-workload-socket" pod "33457c12-09bd-4268-83a6-3543bd62be67" (UID: "33457c12-09bd-4268-83a6-3543bd62be67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:50:03.418700 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.418392 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-istio-data" pod "508ddd74-da86-4674-ad80-7af4a1eb7310" (UID: "508ddd74-da86-4674-ad80-7af4a1eb7310"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:50:03.418700 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.418474 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-workload-certs" pod "33457c12-09bd-4268-83a6-3543bd62be67" (UID: "33457c12-09bd-4268-83a6-3543bd62be67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:50:03.418700 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.418524 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-credential-socket" pod "33457c12-09bd-4268-83a6-3543bd62be67" (UID: "33457c12-09bd-4268-83a6-3543bd62be67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:50:03.419521 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.419229 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33457c12-09bd-4268-83a6-3543bd62be67-istio-token" pod "33457c12-09bd-4268-83a6-3543bd62be67" (UID: "33457c12-09bd-4268-83a6-3543bd62be67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:50:03.420262 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.420236 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/508ddd74-da86-4674-ad80-7af4a1eb7310-istio-podinfo" pod "508ddd74-da86-4674-ad80-7af4a1eb7310" (UID: "508ddd74-da86-4674-ad80-7af4a1eb7310"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:50:03.420366 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.420270 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/508ddd74-da86-4674-ad80-7af4a1eb7310-istio-token" pod "508ddd74-da86-4674-ad80-7af4a1eb7310" (UID: "508ddd74-da86-4674-ad80-7af4a1eb7310"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:50:03.420438 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.420362 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/508ddd74-da86-4674-ad80-7af4a1eb7310-kube-api-access-xvzps" pod "508ddd74-da86-4674-ad80-7af4a1eb7310" (UID: "508ddd74-da86-4674-ad80-7af4a1eb7310"). InnerVolumeSpecName "kube-api-access-xvzps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:50:03.420657 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.420633 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33457c12-09bd-4268-83a6-3543bd62be67-istio-podinfo" pod "33457c12-09bd-4268-83a6-3543bd62be67" (UID: "33457c12-09bd-4268-83a6-3543bd62be67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:50:03.421050 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.421033 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-istio-envoy" pod "508ddd74-da86-4674-ad80-7af4a1eb7310" (UID: "508ddd74-da86-4674-ad80-7af4a1eb7310"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:50:03.422220 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.422203 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-istio-envoy" pod "33457c12-09bd-4268-83a6-3543bd62be67" (UID: "33457c12-09bd-4268-83a6-3543bd62be67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:50:03.517737 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.517660 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/508ddd74-da86-4674-ad80-7af4a1eb7310-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:50:03.517737 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.517690 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:50:03.517737 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.517700 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:50:03.517737 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.517723 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:50:03.517737 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.517733 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:50:03.517737 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.517742 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/508ddd74-da86-4674-ad80-7af4a1eb7310-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:50:03.518088 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.517750 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:50:03.518088 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.517759 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33457c12-09bd-4268-83a6-3543bd62be67-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:50:03.518088 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.517767 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd9q9\" (UniqueName: \"kubernetes.io/projected/33457c12-09bd-4268-83a6-3543bd62be67-kube-api-access-gd9q9\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:50:03.518088 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.517777 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvzps\" (UniqueName: \"kubernetes.io/projected/508ddd74-da86-4674-ad80-7af4a1eb7310-kube-api-access-xvzps\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:50:03.518088 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.517785 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33457c12-09bd-4268-83a6-3543bd62be67-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:50:03.518088 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.517793 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:50:03.518088 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.517801 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:50:03.518088 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.517809 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/508ddd74-da86-4674-ad80-7af4a1eb7310-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:50:03.518088 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.517817 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/508ddd74-da86-4674-ad80-7af4a1eb7310-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:50:03.518088 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:03.517826 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33457c12-09bd-4268-83a6-3543bd62be67-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:50:04.029629 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:04.029602 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" Oct 30 17:50:04.030137 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:04.029640 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh" event={"ID":"33457c12-09bd-4268-83a6-3543bd62be67","Type":"ContainerDied","Data":"c41adceba9ad8be727be32a0b7b38d9c48c092686f4bcbc511d99e2b0eb85bdb"} Oct 30 17:50:04.030137 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:04.029710 4965 scope.go:122] "RemoveContainer" containerID="edb8ac61d2c3c719c9905e8c1632468fdad0efd0aa0daa0a05efb58ce85a3fd0" Oct 30 17:50:04.031311 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:04.031287 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" event={"ID":"508ddd74-da86-4674-ad80-7af4a1eb7310","Type":"ContainerDied","Data":"75b1d603821925e8c78af06cfcd168e3c72b7f23907629d4d37582217cb36f92"} Oct 30 17:50:04.031431 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:04.031321 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g" Oct 30 17:50:04.046375 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:04.046351 4965 scope.go:122] "RemoveContainer" containerID="f294af8eb72ab92728967dc99939fd78b0d055ab55dbe21e483a3559cdc0f587" Oct 30 17:50:04.101712 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:04.101674 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g"] Oct 30 17:50:04.114904 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:04.112238 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-h4d7g"] Oct 30 17:50:04.136489 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:04.136448 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh"] Oct 30 17:50:04.140327 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:04.140300 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-6qxlh"] Oct 30 17:50:05.997624 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:50:05.997586 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:50:06.100959 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:06.100923 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33457c12-09bd-4268-83a6-3543bd62be67" path="/var/lib/kubelet/pods/33457c12-09bd-4268-83a6-3543bd62be67/volumes" Oct 30 17:50:06.101515 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:06.101499 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="508ddd74-da86-4674-ad80-7af4a1eb7310" path="/var/lib/kubelet/pods/508ddd74-da86-4674-ad80-7af4a1eb7310/volumes" Oct 30 17:50:09.176504 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:09.176472 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ee9ff3-55d6b46fd7-rvtsx"] Oct 30 17:50:14.186886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:14.186594 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee9ff3-55d6b46fd7-rvtsx" podUID="6d5970a7-8467-4d2f-99ad-c9f6cc55df7d" containerName="payload-pre-processing" containerID="cri-o://fabcbb54022e689537bd0c5d340bda45071abbc6262683c4667194e1edae49d4" gracePeriod=25 Oct 30 17:50:17.799434 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:17.799399 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee9ff3-55d6b46fd7-rvtsx" podUID="6d5970a7-8467-4d2f-99ad-c9f6cc55df7d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:50:25.571758 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:25.571726 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:50:27.830134 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:27.830090 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee9ff3-55d6b46fd7-rvtsx" podUID="6d5970a7-8467-4d2f-99ad-c9f6cc55df7d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:50:34.415691 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:34.415655 4965 generic.go:356] "Generic (PLEG): container finished" podID="6d5970a7-8467-4d2f-99ad-c9f6cc55df7d" containerID="fabcbb54022e689537bd0c5d340bda45071abbc6262683c4667194e1edae49d4" exitCode=0 Oct 30 17:50:34.416125 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:34.415731 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee9ff3-55d6b46fd7-rvtsx" event={"ID":"6d5970a7-8467-4d2f-99ad-c9f6cc55df7d","Type":"ContainerDied","Data":"fabcbb54022e689537bd0c5d340bda45071abbc6262683c4667194e1edae49d4"} Oct 30 17:50:34.501063 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:34.501039 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee9ff3-55d6b46fd7-rvtsx" Oct 30 17:50:34.577817 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:34.577785 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d5970a7-8467-4d2f-99ad-c9f6cc55df7d-kube-api-access-k4p68\" (UniqueName: \"kubernetes.io/projected/6d5970a7-8467-4d2f-99ad-c9f6cc55df7d-kube-api-access-k4p68\") pod \"6d5970a7-8467-4d2f-99ad-c9f6cc55df7d\" (UID: \"6d5970a7-8467-4d2f-99ad-c9f6cc55df7d\") " Oct 30 17:50:34.578039 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:34.577893 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d5970a7-8467-4d2f-99ad-c9f6cc55df7d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d5970a7-8467-4d2f-99ad-c9f6cc55df7d-plugins-config-volume\") pod \"6d5970a7-8467-4d2f-99ad-c9f6cc55df7d\" (UID: \"6d5970a7-8467-4d2f-99ad-c9f6cc55df7d\") " Oct 30 17:50:34.578389 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:34.578354 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d5970a7-8467-4d2f-99ad-c9f6cc55df7d-plugins-config-volume" pod "6d5970a7-8467-4d2f-99ad-c9f6cc55df7d" (UID: "6d5970a7-8467-4d2f-99ad-c9f6cc55df7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:50:34.578623 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:34.578599 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d5970a7-8467-4d2f-99ad-c9f6cc55df7d-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:50:34.580040 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:34.580012 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d5970a7-8467-4d2f-99ad-c9f6cc55df7d-kube-api-access-k4p68" pod "6d5970a7-8467-4d2f-99ad-c9f6cc55df7d" (UID: "6d5970a7-8467-4d2f-99ad-c9f6cc55df7d"). InnerVolumeSpecName "kube-api-access-k4p68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:50:34.680311 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:34.680269 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4p68\" (UniqueName: \"kubernetes.io/projected/6d5970a7-8467-4d2f-99ad-c9f6cc55df7d-kube-api-access-k4p68\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:50:35.430711 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:35.430673 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee9ff3-55d6b46fd7-rvtsx" event={"ID":"6d5970a7-8467-4d2f-99ad-c9f6cc55df7d","Type":"ContainerDied","Data":"1c4086a117c37af008cdcb13f7c4957fcfea709446ade4eb9fd2ac40c771c82a"} Oct 30 17:50:35.431443 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:35.431408 4965 scope.go:122] "RemoveContainer" containerID="fabcbb54022e689537bd0c5d340bda45071abbc6262683c4667194e1edae49d4" Oct 30 17:50:35.440419 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:35.440391 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee9ff3-55d6b46fd7-rvtsx" Oct 30 17:50:35.534621 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:35.534586 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ee9ff3-55d6b46fd7-rvtsx"] Oct 30 17:50:35.546689 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:35.546660 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ee9ff3-55d6b46fd7-rvtsx"] Oct 30 17:50:35.998069 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:50:35.998032 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:50:36.101277 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:36.101240 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d5970a7-8467-4d2f-99ad-c9f6cc55df7d" path="/var/lib/kubelet/pods/6d5970a7-8467-4d2f-99ad-c9f6cc55df7d/volumes" Oct 30 17:50:38.568817 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:38.568776 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv"] Oct 30 17:50:38.569328 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:38.569108 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv" podUID="0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001" containerName="maas-api" containerID="cri-o://8b7d65e75e1c6837b6b492b60da543b38e3eafda2dfefd3841ad534038337b79" gracePeriod=30 Oct 30 17:50:39.209921 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:39.209499 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv" Oct 30 17:50:39.389824 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:39.389790 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001-maas-api-tls\") pod \"0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001\" (UID: \"0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001\") " Oct 30 17:50:39.389824 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:39.389826 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001-openshift-service-ca\") pod \"0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001\" (UID: \"0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001\") " Oct 30 17:50:39.390111 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:39.389983 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001-metrics-certs\") pod \"0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001\" (UID: \"0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001\") " Oct 30 17:50:39.390177 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:39.390133 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001-kube-api-access-sfkvr\" (UniqueName: \"kubernetes.io/projected/0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001-kube-api-access-sfkvr\") pod \"0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001\" (UID: \"0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001\") " Oct 30 17:50:39.390368 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:39.390336 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001-openshift-service-ca" pod "0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001" (UID: "0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:50:39.391056 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:39.391029 4965 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001-openshift-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:50:39.393039 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:39.393009 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001-metrics-certs" pod "0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001" (UID: "0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:50:39.393154 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:39.393020 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001-maas-api-tls" pod "0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001" (UID: "0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:50:39.393154 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:39.393061 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001-kube-api-access-sfkvr" pod "0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001" (UID: "0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001"). InnerVolumeSpecName "kube-api-access-sfkvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:50:39.504761 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:39.504726 4965 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001-metrics-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:50:39.505061 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:39.505040 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfkvr\" (UniqueName: \"kubernetes.io/projected/0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001-kube-api-access-sfkvr\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:50:39.505199 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:39.505185 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001-maas-api-tls\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:50:39.556034 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:39.555110 4965 generic.go:356] "Generic (PLEG): container finished" podID="0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001" containerID="8b7d65e75e1c6837b6b492b60da543b38e3eafda2dfefd3841ad534038337b79" exitCode=0 Oct 30 17:50:39.556034 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:39.555334 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv" event={"ID":"0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001","Type":"ContainerDied","Data":"8b7d65e75e1c6837b6b492b60da543b38e3eafda2dfefd3841ad534038337b79"} Oct 30 17:50:39.556034 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:39.555387 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv" event={"ID":"0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001","Type":"ContainerDied","Data":"36ffd51fd5f3a07ad4a6a1aee8052198ca0cc7d2ee7f02d371cdf917edda95be"} Oct 30 17:50:39.556034 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:39.555416 4965 scope.go:122] "RemoveContainer" containerID="8b7d65e75e1c6837b6b492b60da543b38e3eafda2dfefd3841ad534038337b79" Oct 30 17:50:39.556034 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:39.555414 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv" Oct 30 17:50:39.570847 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:39.570700 4965 scope.go:122] "RemoveContainer" containerID="8b7d65e75e1c6837b6b492b60da543b38e3eafda2dfefd3841ad534038337b79" Oct 30 17:50:39.571260 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:50:39.571222 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b7d65e75e1c6837b6b492b60da543b38e3eafda2dfefd3841ad534038337b79\": container with ID starting with 8b7d65e75e1c6837b6b492b60da543b38e3eafda2dfefd3841ad534038337b79 not found: ID does not exist" containerID="8b7d65e75e1c6837b6b492b60da543b38e3eafda2dfefd3841ad534038337b79" Oct 30 17:50:39.571451 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:39.571270 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b7d65e75e1c6837b6b492b60da543b38e3eafda2dfefd3841ad534038337b79"} err="failed to get container status \"8b7d65e75e1c6837b6b492b60da543b38e3eafda2dfefd3841ad534038337b79\": rpc error: code = NotFound desc = could not find container \"8b7d65e75e1c6837b6b492b60da543b38e3eafda2dfefd3841ad534038337b79\": container with ID starting with 8b7d65e75e1c6837b6b492b60da543b38e3eafda2dfefd3841ad534038337b79 not found: ID does not exist" Oct 30 17:50:39.636835 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:39.636799 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv"] Oct 30 17:50:39.657658 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:39.657576 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f8b48d-6f798cb5d5-npcbv"] Oct 30 17:50:40.100776 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:40.100744 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001" path="/var/lib/kubelet/pods/0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001/volumes" Oct 30 17:50:41.925969 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:41.925938 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2"] Oct 30 17:50:56.948516 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:56.948478 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" podUID="4ea02e31-5a9d-4070-8227-4e71fb8e849e" containerName="main" containerID="cri-o://5054452fb18603905b533e7a3bfc328f4188eddd9560102cf503282b02b97a66" gracePeriod=45 Oct 30 17:50:57.335182 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:57.335152 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" Oct 30 17:50:57.397500 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:57.397467 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ea02e31-5a9d-4070-8227-4e71fb8e849e-tls-certs\" (UniqueName: \"kubernetes.io/secret/4ea02e31-5a9d-4070-8227-4e71fb8e849e-tls-certs\") pod \"4ea02e31-5a9d-4070-8227-4e71fb8e849e\" (UID: \"4ea02e31-5a9d-4070-8227-4e71fb8e849e\") " Oct 30 17:50:57.397685 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:57.397648 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ea02e31-5a9d-4070-8227-4e71fb8e849e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea02e31-5a9d-4070-8227-4e71fb8e849e-tmp-dir\") pod \"4ea02e31-5a9d-4070-8227-4e71fb8e849e\" (UID: \"4ea02e31-5a9d-4070-8227-4e71fb8e849e\") " Oct 30 17:50:57.397971 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:57.397940 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea02e31-5a9d-4070-8227-4e71fb8e849e-tmp-dir" pod "4ea02e31-5a9d-4070-8227-4e71fb8e849e" (UID: "4ea02e31-5a9d-4070-8227-4e71fb8e849e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:50:57.398109 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:57.398082 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ea02e31-5a9d-4070-8227-4e71fb8e849e-kube-api-access-bfvnr\" (UniqueName: \"kubernetes.io/projected/4ea02e31-5a9d-4070-8227-4e71fb8e849e-kube-api-access-bfvnr\") pod \"4ea02e31-5a9d-4070-8227-4e71fb8e849e\" (UID: \"4ea02e31-5a9d-4070-8227-4e71fb8e849e\") " Oct 30 17:50:57.398526 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:57.398462 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ea02e31-5a9d-4070-8227-4e71fb8e849e-home\" (UniqueName: \"kubernetes.io/empty-dir/4ea02e31-5a9d-4070-8227-4e71fb8e849e-home\") pod \"4ea02e31-5a9d-4070-8227-4e71fb8e849e\" (UID: \"4ea02e31-5a9d-4070-8227-4e71fb8e849e\") " Oct 30 17:50:57.398632 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:57.398559 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ea02e31-5a9d-4070-8227-4e71fb8e849e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ea02e31-5a9d-4070-8227-4e71fb8e849e-model-cache\") pod \"4ea02e31-5a9d-4070-8227-4e71fb8e849e\" (UID: \"4ea02e31-5a9d-4070-8227-4e71fb8e849e\") " Oct 30 17:50:57.398696 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:57.398604 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ea02e31-5a9d-4070-8227-4e71fb8e849e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ea02e31-5a9d-4070-8227-4e71fb8e849e-dshm\") pod \"4ea02e31-5a9d-4070-8227-4e71fb8e849e\" (UID: \"4ea02e31-5a9d-4070-8227-4e71fb8e849e\") " Oct 30 17:50:57.399070 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:57.399024 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea02e31-5a9d-4070-8227-4e71fb8e849e-home" pod "4ea02e31-5a9d-4070-8227-4e71fb8e849e" (UID: "4ea02e31-5a9d-4070-8227-4e71fb8e849e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:50:57.399070 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:57.399042 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea02e31-5a9d-4070-8227-4e71fb8e849e-model-cache" pod "4ea02e31-5a9d-4070-8227-4e71fb8e849e" (UID: "4ea02e31-5a9d-4070-8227-4e71fb8e849e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:50:57.399631 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:57.399603 4965 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ea02e31-5a9d-4070-8227-4e71fb8e849e-home\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:50:57.399631 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:57.399630 4965 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ea02e31-5a9d-4070-8227-4e71fb8e849e-model-cache\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:50:57.399790 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:57.399643 4965 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea02e31-5a9d-4070-8227-4e71fb8e849e-tmp-dir\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:50:57.400058 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:57.400033 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea02e31-5a9d-4070-8227-4e71fb8e849e-tls-certs" pod "4ea02e31-5a9d-4070-8227-4e71fb8e849e" (UID: "4ea02e31-5a9d-4070-8227-4e71fb8e849e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:50:57.402314 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:57.401225 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea02e31-5a9d-4070-8227-4e71fb8e849e-kube-api-access-bfvnr" pod "4ea02e31-5a9d-4070-8227-4e71fb8e849e" (UID: "4ea02e31-5a9d-4070-8227-4e71fb8e849e"). InnerVolumeSpecName "kube-api-access-bfvnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:50:57.404242 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:57.404218 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea02e31-5a9d-4070-8227-4e71fb8e849e-dshm" pod "4ea02e31-5a9d-4070-8227-4e71fb8e849e" (UID: "4ea02e31-5a9d-4070-8227-4e71fb8e849e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:50:57.500725 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:57.500694 4965 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ea02e31-5a9d-4070-8227-4e71fb8e849e-dshm\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:50:57.500725 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:57.500720 4965 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ea02e31-5a9d-4070-8227-4e71fb8e849e-tls-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:50:57.500725 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:57.500731 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfvnr\" (UniqueName: \"kubernetes.io/projected/4ea02e31-5a9d-4070-8227-4e71fb8e849e-kube-api-access-bfvnr\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:50:57.621914 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:57.621856 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt"] Oct 30 17:50:57.952281 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:57.952195 4965 generic.go:356] "Generic (PLEG): container finished" podID="4ea02e31-5a9d-4070-8227-4e71fb8e849e" containerID="5054452fb18603905b533e7a3bfc328f4188eddd9560102cf503282b02b97a66" exitCode=0 Oct 30 17:50:57.952281 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:57.952252 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" Oct 30 17:50:57.952799 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:57.952249 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" event={"ID":"4ea02e31-5a9d-4070-8227-4e71fb8e849e","Type":"ContainerDied","Data":"5054452fb18603905b533e7a3bfc328f4188eddd9560102cf503282b02b97a66"} Oct 30 17:50:57.952799 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:57.952374 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" event={"ID":"4ea02e31-5a9d-4070-8227-4e71fb8e849e","Type":"ContainerDied","Data":"48be7d1feddca4a9b74eb2c8d27ba22f7f7e8a4f5d0f1513d06782788868625c"} Oct 30 17:50:57.952799 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:57.952395 4965 scope.go:122] "RemoveContainer" containerID="5054452fb18603905b533e7a3bfc328f4188eddd9560102cf503282b02b97a66" Oct 30 17:50:57.965203 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:57.965179 4965 scope.go:122] "RemoveContainer" containerID="5054452fb18603905b533e7a3bfc328f4188eddd9560102cf503282b02b97a66" Oct 30 17:50:57.965533 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:50:57.965512 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5054452fb18603905b533e7a3bfc328f4188eddd9560102cf503282b02b97a66\": container with ID starting with 5054452fb18603905b533e7a3bfc328f4188eddd9560102cf503282b02b97a66 not found: ID does not exist" containerID="5054452fb18603905b533e7a3bfc328f4188eddd9560102cf503282b02b97a66" Oct 30 17:50:57.965621 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:57.965546 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5054452fb18603905b533e7a3bfc328f4188eddd9560102cf503282b02b97a66"} err="failed to get container status \"5054452fb18603905b533e7a3bfc328f4188eddd9560102cf503282b02b97a66\": rpc error: code = NotFound desc = could not find container \"5054452fb18603905b533e7a3bfc328f4188eddd9560102cf503282b02b97a66\": container with ID starting with 5054452fb18603905b533e7a3bfc328f4188eddd9560102cf503282b02b97a66 not found: ID does not exist" Oct 30 17:50:58.005909 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:58.005880 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2"] Oct 30 17:50:58.009647 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:58.009621 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2"] Oct 30 17:50:58.100815 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:58.100785 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ea02e31-5a9d-4070-8227-4e71fb8e849e" path="/var/lib/kubelet/pods/4ea02e31-5a9d-4070-8227-4e71fb8e849e/volumes" Oct 30 17:50:58.196992 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:50:58.196947 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-38f445/test-model-38f445-kserve-6d7ff8c99f-nt8g2" podUID="4ea02e31-5a9d-4070-8227-4e71fb8e849e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.154:8000/ready\": context deadline exceeded" Oct 30 17:51:02.622831 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:02.622789 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt" podUID="57ba78c3-6a27-4669-8182-35287812b3a1" containerName="payload-pre-processing" containerID="cri-o://31132fa8b235d7674e9303750a32e2d9a02c2611b55d17fa1f6d64d9ee464bf3" gracePeriod=25 Oct 30 17:51:02.973953 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:02.973846 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt" podUID="57ba78c3-6a27-4669-8182-35287812b3a1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:51:05.901750 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:05.901712 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-01b925-558dd79655-t7zv2"] Oct 30 17:51:05.902512 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:05.902495 4965 cpu_manager.go:414] "containerMap: removing container" podUID="33457c12-09bd-4268-83a6-3543bd62be67" containerName="istio-proxy" Oct 30 17:51:05.902512 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:05.902512 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33457c12-09bd-4268-83a6-3543bd62be67" containerName="istio-proxy" Oct 30 17:51:05.902593 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:05.902532 4965 cpu_manager.go:414] "containerMap: removing container" podUID="4ea02e31-5a9d-4070-8227-4e71fb8e849e" containerName="main" Oct 30 17:51:05.902593 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:05.902537 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea02e31-5a9d-4070-8227-4e71fb8e849e" containerName="main" Oct 30 17:51:05.902593 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:05.902565 4965 cpu_manager.go:414] "containerMap: removing container" podUID="cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca" containerName="maas-api" Oct 30 17:51:05.902593 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:05.902571 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca" containerName="maas-api" Oct 30 17:51:05.902593 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:05.902580 4965 cpu_manager.go:414] "containerMap: removing container" podUID="6d5970a7-8467-4d2f-99ad-c9f6cc55df7d" containerName="payload-pre-processing" Oct 30 17:51:05.902593 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:05.902585 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d5970a7-8467-4d2f-99ad-c9f6cc55df7d" containerName="payload-pre-processing" Oct 30 17:51:05.902772 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:05.902597 4965 cpu_manager.go:414] "containerMap: removing container" podUID="0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001" containerName="maas-api" Oct 30 17:51:05.902772 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:05.902602 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001" containerName="maas-api" Oct 30 17:51:05.902772 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:05.902615 4965 cpu_manager.go:414] "containerMap: removing container" podUID="508ddd74-da86-4674-ad80-7af4a1eb7310" containerName="istio-proxy" Oct 30 17:51:05.902772 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:05.902619 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="508ddd74-da86-4674-ad80-7af4a1eb7310" containerName="istio-proxy" Oct 30 17:51:05.902772 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:05.902627 4965 cpu_manager.go:414] "containerMap: removing container" podUID="508ddd74-da86-4674-ad80-7af4a1eb7310" containerName="istio-proxy" Oct 30 17:51:05.902772 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:05.902632 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="508ddd74-da86-4674-ad80-7af4a1eb7310" containerName="istio-proxy" Oct 30 17:51:05.902772 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:05.902745 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="508ddd74-da86-4674-ad80-7af4a1eb7310" containerName="istio-proxy" Oct 30 17:51:05.902772 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:05.902758 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="33457c12-09bd-4268-83a6-3543bd62be67" containerName="istio-proxy" Oct 30 17:51:05.902772 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:05.902765 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea02e31-5a9d-4070-8227-4e71fb8e849e" containerName="main" Oct 30 17:51:05.903075 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:05.902776 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf2a6d30-2a2c-4ee2-a4ad-18d917a634ca" containerName="maas-api" Oct 30 17:51:05.903075 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:05.902786 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d5970a7-8467-4d2f-99ad-c9f6cc55df7d" containerName="payload-pre-processing" Oct 30 17:51:05.903075 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:05.902792 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c26ab04-4bf8-4f0f-bb42-5fcb6b8f5001" containerName="maas-api" Oct 30 17:51:05.908079 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:05.908062 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-01b925-558dd79655-t7zv2" Oct 30 17:51:05.926989 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:05.926961 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-01b925-558dd79655-t7zv2"] Oct 30 17:51:05.932702 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:05.932676 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-01b925-dockercfg-zj4gz\"" Oct 30 17:51:06.011283 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:06.011241 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-01b925\"" Oct 30 17:51:06.253988 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:06.251752 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24f2394c-1e9c-417e-91c0-5086ebae37c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-01b925-558dd79655-t7zv2\" (UID: \"24f2394c-1e9c-417e-91c0-5086ebae37c0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-01b925-558dd79655-t7zv2" Oct 30 17:51:06.253988 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:06.252647 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hnz6\" (UniqueName: \"kubernetes.io/projected/24f2394c-1e9c-417e-91c0-5086ebae37c0-kube-api-access-6hnz6\") pod \"payload-pre-processing-e2e-shared-b-w1-01b925-558dd79655-t7zv2\" (UID: \"24f2394c-1e9c-417e-91c0-5086ebae37c0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-01b925-558dd79655-t7zv2" Oct 30 17:51:06.294109 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:51:06.294074 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:51:06.354481 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:06.354444 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24f2394c-1e9c-417e-91c0-5086ebae37c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-01b925-558dd79655-t7zv2\" (UID: \"24f2394c-1e9c-417e-91c0-5086ebae37c0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-01b925-558dd79655-t7zv2" Oct 30 17:51:06.354650 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:06.354577 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hnz6\" (UniqueName: \"kubernetes.io/projected/24f2394c-1e9c-417e-91c0-5086ebae37c0-kube-api-access-6hnz6\") pod \"payload-pre-processing-e2e-shared-b-w1-01b925-558dd79655-t7zv2\" (UID: \"24f2394c-1e9c-417e-91c0-5086ebae37c0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-01b925-558dd79655-t7zv2" Oct 30 17:51:06.355282 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:06.355263 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24f2394c-1e9c-417e-91c0-5086ebae37c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-01b925-558dd79655-t7zv2\" (UID: \"24f2394c-1e9c-417e-91c0-5086ebae37c0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-01b925-558dd79655-t7zv2" Oct 30 17:51:06.381217 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:06.381186 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hnz6\" (UniqueName: \"kubernetes.io/projected/24f2394c-1e9c-417e-91c0-5086ebae37c0-kube-api-access-6hnz6\") pod \"payload-pre-processing-e2e-shared-b-w1-01b925-558dd79655-t7zv2\" (UID: \"24f2394c-1e9c-417e-91c0-5086ebae37c0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-01b925-558dd79655-t7zv2" Oct 30 17:51:06.525062 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:06.524975 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-01b925-558dd79655-t7zv2" Oct 30 17:51:06.852503 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:06.852472 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-01b925-558dd79655-t7zv2"] Oct 30 17:51:06.853691 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:51:06.853664 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f2394c_1e9c_417e_91c0_5086ebae37c0.slice/crio-0a63395915398e913326262ec8b96559ff4c03300d59a2e3a5ba41df997d71f5 WatchSource:0}: Error finding container 0a63395915398e913326262ec8b96559ff4c03300d59a2e3a5ba41df997d71f5: Status 404 returned error can't find the container with id 0a63395915398e913326262ec8b96559ff4c03300d59a2e3a5ba41df997d71f5 Oct 30 17:51:07.160610 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:07.160529 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-01b925-558dd79655-t7zv2" event={"ID":"24f2394c-1e9c-417e-91c0-5086ebae37c0","Type":"ContainerStarted","Data":"0a6fea3abed593fc598c1a02f98fd47de968da77b1d59af06db74e66f05237e5"} Oct 30 17:51:07.160610 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:07.160593 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-01b925-558dd79655-t7zv2" event={"ID":"24f2394c-1e9c-417e-91c0-5086ebae37c0","Type":"ContainerStarted","Data":"0a63395915398e913326262ec8b96559ff4c03300d59a2e3a5ba41df997d71f5"} Oct 30 17:51:07.161045 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:07.160745 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-01b925-558dd79655-t7zv2" Oct 30 17:51:07.209188 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:07.209128 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-01b925-558dd79655-t7zv2" podStartSLOduration=2.209110327 podStartE2EDuration="2.209110327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:51:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:07.200578329 +0000 UTC m=+2325.252678272" watchObservedRunningTime="2026-10-30 17:51:07.209110327 +0000 UTC m=+2325.261210262" Oct 30 17:51:12.973341 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:12.973306 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt" podUID="57ba78c3-6a27-4669-8182-35287812b3a1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:51:14.681053 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:14.681019 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx"] Oct 30 17:51:14.682550 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:14.682523 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="508ddd74-da86-4674-ad80-7af4a1eb7310" containerName="istio-proxy" Oct 30 17:51:14.687455 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:14.687430 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" Oct 30 17:51:14.707855 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:14.707826 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx"] Oct 30 17:51:14.710535 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:14.710506 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-c31c73\"/\"kube-root-ca.crt\"" Oct 30 17:51:14.710680 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:14.710618 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-c31c73\"/\"openshift-service-ca.crt\"" Oct 30 17:51:14.713298 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:14.713274 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-c31c73\"/\"default-dockercfg-m42t8\"" Oct 30 17:51:14.720004 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:14.719971 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-c31c73\"/\"auth-test-model-c31c73-kserve-self-signed-certs\"" Oct 30 17:51:14.796636 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:14.796603 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mccz9\" (UniqueName: \"kubernetes.io/projected/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-kube-api-access-mccz9\") pod \"auth-test-model-c31c73-kserve-594654b985-cxdkx\" (UID: \"7bd734c0-9b03-463b-adfd-e1a206fd1f9a\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" Oct 30 17:51:14.796802 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:14.796647 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-tmp-dir\") pod \"auth-test-model-c31c73-kserve-594654b985-cxdkx\" (UID: \"7bd734c0-9b03-463b-adfd-e1a206fd1f9a\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" Oct 30 17:51:14.796802 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:14.796736 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-dshm\") pod \"auth-test-model-c31c73-kserve-594654b985-cxdkx\" (UID: \"7bd734c0-9b03-463b-adfd-e1a206fd1f9a\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" Oct 30 17:51:14.796903 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:14.796830 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-home\") pod \"auth-test-model-c31c73-kserve-594654b985-cxdkx\" (UID: \"7bd734c0-9b03-463b-adfd-e1a206fd1f9a\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" Oct 30 17:51:14.796990 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:14.796970 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-model-cache\") pod \"auth-test-model-c31c73-kserve-594654b985-cxdkx\" (UID: \"7bd734c0-9b03-463b-adfd-e1a206fd1f9a\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" Oct 30 17:51:14.797050 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:14.797039 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-tls-certs\") pod \"auth-test-model-c31c73-kserve-594654b985-cxdkx\" (UID: \"7bd734c0-9b03-463b-adfd-e1a206fd1f9a\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" Oct 30 17:51:14.899205 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:14.899172 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mccz9\" (UniqueName: \"kubernetes.io/projected/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-kube-api-access-mccz9\") pod \"auth-test-model-c31c73-kserve-594654b985-cxdkx\" (UID: \"7bd734c0-9b03-463b-adfd-e1a206fd1f9a\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" Oct 30 17:51:14.899386 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:14.899215 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-tmp-dir\") pod \"auth-test-model-c31c73-kserve-594654b985-cxdkx\" (UID: \"7bd734c0-9b03-463b-adfd-e1a206fd1f9a\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" Oct 30 17:51:14.899386 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:14.899245 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-dshm\") pod \"auth-test-model-c31c73-kserve-594654b985-cxdkx\" (UID: \"7bd734c0-9b03-463b-adfd-e1a206fd1f9a\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" Oct 30 17:51:14.899386 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:14.899282 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-home\") pod \"auth-test-model-c31c73-kserve-594654b985-cxdkx\" (UID: \"7bd734c0-9b03-463b-adfd-e1a206fd1f9a\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" Oct 30 17:51:14.899386 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:14.899337 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-model-cache\") pod \"auth-test-model-c31c73-kserve-594654b985-cxdkx\" (UID: \"7bd734c0-9b03-463b-adfd-e1a206fd1f9a\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" Oct 30 17:51:14.899386 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:14.899384 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-tls-certs\") pod \"auth-test-model-c31c73-kserve-594654b985-cxdkx\" (UID: \"7bd734c0-9b03-463b-adfd-e1a206fd1f9a\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" Oct 30 17:51:14.899715 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:14.899691 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-tmp-dir\") pod \"auth-test-model-c31c73-kserve-594654b985-cxdkx\" (UID: \"7bd734c0-9b03-463b-adfd-e1a206fd1f9a\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" Oct 30 17:51:14.899816 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:14.899766 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-home\") pod \"auth-test-model-c31c73-kserve-594654b985-cxdkx\" (UID: \"7bd734c0-9b03-463b-adfd-e1a206fd1f9a\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" Oct 30 17:51:14.899908 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:14.899811 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-model-cache\") pod \"auth-test-model-c31c73-kserve-594654b985-cxdkx\" (UID: \"7bd734c0-9b03-463b-adfd-e1a206fd1f9a\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" Oct 30 17:51:14.901735 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:14.901714 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-dshm\") pod \"auth-test-model-c31c73-kserve-594654b985-cxdkx\" (UID: \"7bd734c0-9b03-463b-adfd-e1a206fd1f9a\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" Oct 30 17:51:14.901901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:14.901885 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-tls-certs\") pod \"auth-test-model-c31c73-kserve-594654b985-cxdkx\" (UID: \"7bd734c0-9b03-463b-adfd-e1a206fd1f9a\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" Oct 30 17:51:14.922219 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:14.922195 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mccz9\" (UniqueName: \"kubernetes.io/projected/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-kube-api-access-mccz9\") pod \"auth-test-model-c31c73-kserve-594654b985-cxdkx\" (UID: \"7bd734c0-9b03-463b-adfd-e1a206fd1f9a\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" Oct 30 17:51:15.037464 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:15.037382 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" Oct 30 17:51:15.403926 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:15.403895 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx"] Oct 30 17:51:15.404110 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:51:15.404089 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bd734c0_9b03_463b_adfd_e1a206fd1f9a.slice/crio-6613d5dcdb18b5d932764119b23d4e8084892ab0672b29e04500e0db3692e4c0 WatchSource:0}: Error finding container 6613d5dcdb18b5d932764119b23d4e8084892ab0672b29e04500e0db3692e4c0: Status 404 returned error can't find the container with id 6613d5dcdb18b5d932764119b23d4e8084892ab0672b29e04500e0db3692e4c0 Oct 30 17:51:16.246539 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:16.237316 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" event={"ID":"7bd734c0-9b03-463b-adfd-e1a206fd1f9a","Type":"ContainerStarted","Data":"c42c009df56ffbc9b72a033d306165886a91f90bd6ebf97edba95768b120f4c6"} Oct 30 17:51:16.246539 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:16.237374 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" event={"ID":"7bd734c0-9b03-463b-adfd-e1a206fd1f9a","Type":"ContainerStarted","Data":"6613d5dcdb18b5d932764119b23d4e8084892ab0672b29e04500e0db3692e4c0"} Oct 30 17:51:17.037939 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:17.037905 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" Oct 30 17:51:18.172669 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:18.172636 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-01b925-558dd79655-t7zv2" Oct 30 17:51:18.216654 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:18.216586 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" podStartSLOduration=4.216571722 podStartE2EDuration="4.216571722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:16.292089198 +0000 UTC m=+2334.344189152" watchObservedRunningTime="2026-10-30 17:51:18.216571722 +0000 UTC m=+2336.268671652" Oct 30 17:51:23.130108 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:23.130075 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt" Oct 30 17:51:23.324000 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:23.323904 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57ba78c3-6a27-4669-8182-35287812b3a1-kube-api-access-dkj6b\" (UniqueName: \"kubernetes.io/projected/57ba78c3-6a27-4669-8182-35287812b3a1-kube-api-access-dkj6b\") pod \"57ba78c3-6a27-4669-8182-35287812b3a1\" (UID: \"57ba78c3-6a27-4669-8182-35287812b3a1\") " Oct 30 17:51:23.324207 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:23.324019 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57ba78c3-6a27-4669-8182-35287812b3a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57ba78c3-6a27-4669-8182-35287812b3a1-plugins-config-volume\") pod \"57ba78c3-6a27-4669-8182-35287812b3a1\" (UID: \"57ba78c3-6a27-4669-8182-35287812b3a1\") " Oct 30 17:51:23.325739 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:23.325697 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57ba78c3-6a27-4669-8182-35287812b3a1-plugins-config-volume" pod "57ba78c3-6a27-4669-8182-35287812b3a1" (UID: "57ba78c3-6a27-4669-8182-35287812b3a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:51:23.328791 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:23.328767 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57ba78c3-6a27-4669-8182-35287812b3a1-kube-api-access-dkj6b" pod "57ba78c3-6a27-4669-8182-35287812b3a1" (UID: "57ba78c3-6a27-4669-8182-35287812b3a1"). InnerVolumeSpecName "kube-api-access-dkj6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:51:23.425831 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:23.425793 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57ba78c3-6a27-4669-8182-35287812b3a1-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:51:23.425831 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:23.425825 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkj6b\" (UniqueName: \"kubernetes.io/projected/57ba78c3-6a27-4669-8182-35287812b3a1-kube-api-access-dkj6b\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:51:23.450831 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:23.450800 4965 generic.go:356] "Generic (PLEG): container finished" podID="57ba78c3-6a27-4669-8182-35287812b3a1" containerID="31132fa8b235d7674e9303750a32e2d9a02c2611b55d17fa1f6d64d9ee464bf3" exitCode=0 Oct 30 17:51:23.451006 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:23.450963 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt" Oct 30 17:51:23.451054 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:23.451027 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt" event={"ID":"57ba78c3-6a27-4669-8182-35287812b3a1","Type":"ContainerDied","Data":"31132fa8b235d7674e9303750a32e2d9a02c2611b55d17fa1f6d64d9ee464bf3"} Oct 30 17:51:23.451100 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:23.451069 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt" event={"ID":"57ba78c3-6a27-4669-8182-35287812b3a1","Type":"ContainerDied","Data":"49ff056d674833ee9798fb00f1123c130e98fa4fc992380dd84efdeeb1adfad2"} Oct 30 17:51:23.451100 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:23.451096 4965 scope.go:122] "RemoveContainer" containerID="31132fa8b235d7674e9303750a32e2d9a02c2611b55d17fa1f6d64d9ee464bf3" Oct 30 17:51:23.514022 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:23.514004 4965 scope.go:122] "RemoveContainer" containerID="31132fa8b235d7674e9303750a32e2d9a02c2611b55d17fa1f6d64d9ee464bf3" Oct 30 17:51:23.514407 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:51:23.514357 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31132fa8b235d7674e9303750a32e2d9a02c2611b55d17fa1f6d64d9ee464bf3\": container with ID starting with 31132fa8b235d7674e9303750a32e2d9a02c2611b55d17fa1f6d64d9ee464bf3 not found: ID does not exist" containerID="31132fa8b235d7674e9303750a32e2d9a02c2611b55d17fa1f6d64d9ee464bf3" Oct 30 17:51:23.514407 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:23.514388 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31132fa8b235d7674e9303750a32e2d9a02c2611b55d17fa1f6d64d9ee464bf3"} err="failed to get container status \"31132fa8b235d7674e9303750a32e2d9a02c2611b55d17fa1f6d64d9ee464bf3\": rpc error: code = NotFound desc = could not find container \"31132fa8b235d7674e9303750a32e2d9a02c2611b55d17fa1f6d64d9ee464bf3\": container with ID starting with 31132fa8b235d7674e9303750a32e2d9a02c2611b55d17fa1f6d64d9ee464bf3 not found: ID does not exist" Oct 30 17:51:23.551187 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:23.551156 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt"] Oct 30 17:51:23.610728 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:23.610491 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt"] Oct 30 17:51:23.993038 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:23.987031 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fb0d-6c444d8c47-6r6qt" podUID="57ba78c3-6a27-4669-8182-35287812b3a1" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.145:50052\" within 1s: context deadline exceeded" Oct 30 17:51:23.993038 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:51:23.987226 4965 logging.go:55] [core] [Channel #5959 SubChannel #5960]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.145:50052", ServerName: "10.132.0.145:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.145:50052: operation was canceled" Oct 30 17:51:24.106616 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:24.106583 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57ba78c3-6a27-4669-8182-35287812b3a1" path="/var/lib/kubelet/pods/57ba78c3-6a27-4669-8182-35287812b3a1/volumes" Oct 30 17:51:25.037726 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:25.037699 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" Oct 30 17:51:25.051456 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:25.051428 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" Oct 30 17:51:25.503445 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:25.503415 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" Oct 30 17:51:27.224641 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:27.224601 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn"] Oct 30 17:51:27.225279 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:27.224916 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" podUID="cc2fac0f-42be-41e6-8288-cdaaeaab7a1d" containerName="istio-proxy" containerID="cri-o://ce9f8f640e293cde94e421cfa710e735b7d2b251a893e107bb6cab6f6d3f36d1" gracePeriod=30 Oct 30 17:51:28.788325 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:28.788243 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p"] Oct 30 17:51:28.788830 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:28.788522 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p" podUID="9079b50c-c40f-4866-b177-414d6511ec8d" containerName="maas-api" containerID="cri-o://a57e53f8bedeca16ed83fdfad5d3744c11d444aa50ae2c96b5a9d828b347558e" gracePeriod=30 Oct 30 17:51:29.423330 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:29.423080 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p" Oct 30 17:51:29.545588 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:29.545499 4965 generic.go:356] "Generic (PLEG): container finished" podID="9079b50c-c40f-4866-b177-414d6511ec8d" containerID="a57e53f8bedeca16ed83fdfad5d3744c11d444aa50ae2c96b5a9d828b347558e" exitCode=0 Oct 30 17:51:29.545588 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:29.545542 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p" Oct 30 17:51:29.548001 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:29.545586 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p" event={"ID":"9079b50c-c40f-4866-b177-414d6511ec8d","Type":"ContainerDied","Data":"a57e53f8bedeca16ed83fdfad5d3744c11d444aa50ae2c96b5a9d828b347558e"} Oct 30 17:51:29.548001 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:29.545649 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p" event={"ID":"9079b50c-c40f-4866-b177-414d6511ec8d","Type":"ContainerDied","Data":"aa21d5328d41cef33d010d0d86d86c6a14c470c7001a285b9e9ffa284d55a838"} Oct 30 17:51:29.548001 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:29.545676 4965 scope.go:122] "RemoveContainer" containerID="a57e53f8bedeca16ed83fdfad5d3744c11d444aa50ae2c96b5a9d828b347558e" Oct 30 17:51:29.548001 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:29.546044 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9079b50c-c40f-4866-b177-414d6511ec8d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9079b50c-c40f-4866-b177-414d6511ec8d-metrics-certs\") pod \"9079b50c-c40f-4866-b177-414d6511ec8d\" (UID: \"9079b50c-c40f-4866-b177-414d6511ec8d\") " Oct 30 17:51:29.548001 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:29.546164 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9079b50c-c40f-4866-b177-414d6511ec8d-kube-api-access-pfmjw\" (UniqueName: \"kubernetes.io/projected/9079b50c-c40f-4866-b177-414d6511ec8d-kube-api-access-pfmjw\") pod \"9079b50c-c40f-4866-b177-414d6511ec8d\" (UID: \"9079b50c-c40f-4866-b177-414d6511ec8d\") " Oct 30 17:51:29.548001 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:29.546239 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9079b50c-c40f-4866-b177-414d6511ec8d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9079b50c-c40f-4866-b177-414d6511ec8d-maas-api-tls\") pod \"9079b50c-c40f-4866-b177-414d6511ec8d\" (UID: \"9079b50c-c40f-4866-b177-414d6511ec8d\") " Oct 30 17:51:29.548001 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:29.546298 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9079b50c-c40f-4866-b177-414d6511ec8d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9079b50c-c40f-4866-b177-414d6511ec8d-openshift-service-ca\") pod \"9079b50c-c40f-4866-b177-414d6511ec8d\" (UID: \"9079b50c-c40f-4866-b177-414d6511ec8d\") " Oct 30 17:51:29.548001 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:29.546764 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9079b50c-c40f-4866-b177-414d6511ec8d-openshift-service-ca" pod "9079b50c-c40f-4866-b177-414d6511ec8d" (UID: "9079b50c-c40f-4866-b177-414d6511ec8d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:51:29.548001 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:29.547161 4965 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9079b50c-c40f-4866-b177-414d6511ec8d-openshift-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:51:29.548669 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:29.548535 4965 generic.go:356] "Generic (PLEG): container finished" podID="b48e4e05-e95d-424c-887e-668bdc42e398" containerID="c496ac3cc83d4845fddb5f3a7d2d11aba86a1c1693c78d96f222c97494166752" exitCode=0 Oct 30 17:51:29.548669 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:29.548548 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9079b50c-c40f-4866-b177-414d6511ec8d-metrics-certs" pod "9079b50c-c40f-4866-b177-414d6511ec8d" (UID: "9079b50c-c40f-4866-b177-414d6511ec8d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:51:29.548669 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:29.548620 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" event={"ID":"b48e4e05-e95d-424c-887e-668bdc42e398","Type":"ContainerDied","Data":"c496ac3cc83d4845fddb5f3a7d2d11aba86a1c1693c78d96f222c97494166752"} Oct 30 17:51:29.548956 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:29.548920 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9079b50c-c40f-4866-b177-414d6511ec8d-maas-api-tls" pod "9079b50c-c40f-4866-b177-414d6511ec8d" (UID: "9079b50c-c40f-4866-b177-414d6511ec8d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:51:29.549077 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:29.549063 4965 scope.go:122] "RemoveContainer" containerID="c496ac3cc83d4845fddb5f3a7d2d11aba86a1c1693c78d96f222c97494166752" Oct 30 17:51:29.549827 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:29.549805 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9079b50c-c40f-4866-b177-414d6511ec8d-kube-api-access-pfmjw" pod "9079b50c-c40f-4866-b177-414d6511ec8d" (UID: "9079b50c-c40f-4866-b177-414d6511ec8d"). InnerVolumeSpecName "kube-api-access-pfmjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:51:29.551077 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:29.551043 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:51:29.551160 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:29.551121 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:51:29.551204 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:29.551164 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:51:29.557725 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:29.557695 4965 scope.go:122] "RemoveContainer" containerID="a57e53f8bedeca16ed83fdfad5d3744c11d444aa50ae2c96b5a9d828b347558e" Oct 30 17:51:29.558027 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:51:29.558004 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a57e53f8bedeca16ed83fdfad5d3744c11d444aa50ae2c96b5a9d828b347558e\": container with ID starting with a57e53f8bedeca16ed83fdfad5d3744c11d444aa50ae2c96b5a9d828b347558e not found: ID does not exist" containerID="a57e53f8bedeca16ed83fdfad5d3744c11d444aa50ae2c96b5a9d828b347558e" Oct 30 17:51:29.558098 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:29.558035 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a57e53f8bedeca16ed83fdfad5d3744c11d444aa50ae2c96b5a9d828b347558e"} err="failed to get container status \"a57e53f8bedeca16ed83fdfad5d3744c11d444aa50ae2c96b5a9d828b347558e\": rpc error: code = NotFound desc = could not find container \"a57e53f8bedeca16ed83fdfad5d3744c11d444aa50ae2c96b5a9d828b347558e\": container with ID starting with a57e53f8bedeca16ed83fdfad5d3744c11d444aa50ae2c96b5a9d828b347558e not found: ID does not exist" Oct 30 17:51:29.649138 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:29.649101 4965 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9079b50c-c40f-4866-b177-414d6511ec8d-metrics-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:51:29.649138 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:29.649139 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfmjw\" (UniqueName: \"kubernetes.io/projected/9079b50c-c40f-4866-b177-414d6511ec8d-kube-api-access-pfmjw\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:51:29.649392 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:29.649155 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9079b50c-c40f-4866-b177-414d6511ec8d-maas-api-tls\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:51:29.905952 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:51:29.905920 4965 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9079b50c_c40f_4866_b177_414d6511ec8d.slice\": RecentStats: unable to find data in memory cache]" Oct 30 17:51:29.929516 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:29.929488 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p"] Oct 30 17:51:29.936224 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:29.936195 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24fb0d-7d6998bd48-bn44p"] Oct 30 17:51:30.102219 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:30.102177 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9079b50c-c40f-4866-b177-414d6511ec8d" path="/var/lib/kubelet/pods/9079b50c-c40f-4866-b177-414d6511ec8d/volumes" Oct 30 17:51:30.581888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:30.567416 4965 generic.go:356] "Generic (PLEG): container finished" podID="b8cbab33-ea35-4e2e-a82d-74e1f47c5b13" containerID="ff71ab324ee976af17b669afc245f2fc029c7e1dda2cde3f69189ac352495334" exitCode=0 Oct 30 17:51:30.581888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:30.567582 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" event={"ID":"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13","Type":"ContainerDied","Data":"ff71ab324ee976af17b669afc245f2fc029c7e1dda2cde3f69189ac352495334"} Oct 30 17:51:30.581888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:30.568207 4965 scope.go:122] "RemoveContainer" containerID="ff71ab324ee976af17b669afc245f2fc029c7e1dda2cde3f69189ac352495334" Oct 30 17:51:30.581888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:30.574300 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:51:30.581888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:30.574403 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:51:30.581888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:30.574448 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:51:30.585481 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:30.585438 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" event={"ID":"b48e4e05-e95d-424c-887e-668bdc42e398","Type":"ContainerStarted","Data":"b9c4f58f889a7e1a4a99f5438229707c75b52e7b3780b12d599a8f6b27e97649"} Oct 30 17:51:31.382883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:31.382144 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:51:31.382883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:31.382281 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 17:51:31.382883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:31.382332 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" podUID="b48e4e05-e95d-424c-887e-668bdc42e398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 17:51:31.620879 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:31.618772 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" event={"ID":"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13","Type":"ContainerStarted","Data":"dd71ba8c3685642fd40121ff243662611cfedc852dd4644b540f39ab6e9bd42f"} Oct 30 17:51:32.216415 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.216328 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:51:32.218087 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.218059 4965 patch_prober.go:29] interesting pod/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 17:51:32.218221 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.218106 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" podUID="b8cbab33-ea35-4e2e-a82d-74e1f47c5b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 17:51:32.397888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.397543 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 17:51:32.397888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.397611 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" podUID="b48e4e05-e95d-424c-887e-668bdc42e398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 17:51:32.634809 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.634776 4965 generic.go:356] "Generic (PLEG): container finished" podID="cc2fac0f-42be-41e6-8288-cdaaeaab7a1d" containerID="ce9f8f640e293cde94e421cfa710e735b7d2b251a893e107bb6cab6f6d3f36d1" exitCode=0 Oct 30 17:51:32.634975 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.634848 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" event={"ID":"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d","Type":"ContainerDied","Data":"ce9f8f640e293cde94e421cfa710e735b7d2b251a893e107bb6cab6f6d3f36d1"} Oct 30 17:51:32.718345 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.718309 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:51:32.753279 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.753248 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-workload-socket\") pod \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " Oct 30 17:51:32.753578 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.753557 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istiod-ca-cert\") pod \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " Oct 30 17:51:32.753780 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.753763 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istio-envoy\") pod \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " Oct 30 17:51:32.753945 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.753928 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-workload-certs\") pod \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " Oct 30 17:51:32.754181 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.754155 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istio-podinfo\") pod \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " Oct 30 17:51:32.754501 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.754484 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istio-data\") pod \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " Oct 30 17:51:32.779270 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.779240 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-kube-api-access-b2qx9\" (UniqueName: \"kubernetes.io/projected/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-kube-api-access-b2qx9\") pod \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " Oct 30 17:51:32.795082 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.795043 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-credential-socket\") pod \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " Oct 30 17:51:32.826147 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.753952 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-workload-socket" pod "cc2fac0f-42be-41e6-8288-cdaaeaab7a1d" (UID: "cc2fac0f-42be-41e6-8288-cdaaeaab7a1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:32.826351 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.785307 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istiod-ca-cert" pod "cc2fac0f-42be-41e6-8288-cdaaeaab7a1d" (UID: "cc2fac0f-42be-41e6-8288-cdaaeaab7a1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:51:32.826351 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.786876 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istio-data" pod "cc2fac0f-42be-41e6-8288-cdaaeaab7a1d" (UID: "cc2fac0f-42be-41e6-8288-cdaaeaab7a1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:32.826351 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.802137 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-credential-socket" pod "cc2fac0f-42be-41e6-8288-cdaaeaab7a1d" (UID: "cc2fac0f-42be-41e6-8288-cdaaeaab7a1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:32.826351 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.803015 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-workload-certs" pod "cc2fac0f-42be-41e6-8288-cdaaeaab7a1d" (UID: "cc2fac0f-42be-41e6-8288-cdaaeaab7a1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:32.826579 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.826351 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istio-token\" (UniqueName: \"kubernetes.io/projected/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istio-token\") pod \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\" (UID: \"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d\") " Oct 30 17:51:32.827795 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.827764 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:51:32.827975 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.827802 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:51:32.827975 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.827840 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:51:32.827975 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.827853 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:51:32.827975 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.827900 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:51:32.831142 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.831007 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-kube-api-access-b2qx9" pod "cc2fac0f-42be-41e6-8288-cdaaeaab7a1d" (UID: "cc2fac0f-42be-41e6-8288-cdaaeaab7a1d"). InnerVolumeSpecName "kube-api-access-b2qx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:51:32.831142 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.831114 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istio-token" pod "cc2fac0f-42be-41e6-8288-cdaaeaab7a1d" (UID: "cc2fac0f-42be-41e6-8288-cdaaeaab7a1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:51:32.831392 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.831366 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istio-podinfo" pod "cc2fac0f-42be-41e6-8288-cdaaeaab7a1d" (UID: "cc2fac0f-42be-41e6-8288-cdaaeaab7a1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:51:32.831846 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.831824 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istio-envoy" pod "cc2fac0f-42be-41e6-8288-cdaaeaab7a1d" (UID: "cc2fac0f-42be-41e6-8288-cdaaeaab7a1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:32.930418 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.930327 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:51:32.930418 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.930367 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:51:32.930418 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.930383 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2qx9\" (UniqueName: \"kubernetes.io/projected/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-kube-api-access-b2qx9\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:51:32.930418 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:32.930397 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:51:33.216735 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:33.216640 4965 patch_prober.go:29] interesting pod/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 17:51:33.216924 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:33.216728 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" podUID="b8cbab33-ea35-4e2e-a82d-74e1f47c5b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 17:51:33.382223 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:33.382180 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 17:51:33.382439 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:33.382260 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" podUID="b48e4e05-e95d-424c-887e-668bdc42e398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 17:51:33.612092 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:33.612056 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:51:33.643464 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:33.643440 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" Oct 30 17:51:33.643626 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:33.643515 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn" event={"ID":"cc2fac0f-42be-41e6-8288-cdaaeaab7a1d","Type":"ContainerDied","Data":"5f804571e3363e57eda233754d979cbf65f3d00ece531bab16f32d0a830db0ee"} Oct 30 17:51:33.643626 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:33.643564 4965 scope.go:122] "RemoveContainer" containerID="ce9f8f640e293cde94e421cfa710e735b7d2b251a893e107bb6cab6f6d3f36d1" Oct 30 17:51:33.747601 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:33.747565 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn"] Oct 30 17:51:33.751773 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:33.751745 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-blssn"] Oct 30 17:51:34.117291 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.117261 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc2fac0f-42be-41e6-8288-cdaaeaab7a1d" path="/var/lib/kubelet/pods/cc2fac0f-42be-41e6-8288-cdaaeaab7a1d/volumes" Oct 30 17:51:34.216981 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.216943 4965 patch_prober.go:29] interesting pod/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 17:51:34.217155 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.216997 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" podUID="b8cbab33-ea35-4e2e-a82d-74e1f47c5b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 17:51:34.390940 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.387245 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 17:51:34.390940 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.387307 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" podUID="b48e4e05-e95d-424c-887e-668bdc42e398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 17:51:34.605888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.599221 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn"] Oct 30 17:51:34.611076 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.611045 4965 cpu_manager.go:414] "containerMap: removing container" podUID="cc2fac0f-42be-41e6-8288-cdaaeaab7a1d" containerName="istio-proxy" Oct 30 17:51:34.611076 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.611073 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc2fac0f-42be-41e6-8288-cdaaeaab7a1d" containerName="istio-proxy" Oct 30 17:51:34.611336 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.611131 4965 cpu_manager.go:414] "containerMap: removing container" podUID="9079b50c-c40f-4866-b177-414d6511ec8d" containerName="maas-api" Oct 30 17:51:34.611336 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.611138 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9079b50c-c40f-4866-b177-414d6511ec8d" containerName="maas-api" Oct 30 17:51:34.611336 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.611182 4965 cpu_manager.go:414] "containerMap: removing container" podUID="57ba78c3-6a27-4669-8182-35287812b3a1" containerName="payload-pre-processing" Oct 30 17:51:34.611336 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.611189 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57ba78c3-6a27-4669-8182-35287812b3a1" containerName="payload-pre-processing" Oct 30 17:51:34.611542 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.611389 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="9079b50c-c40f-4866-b177-414d6511ec8d" containerName="maas-api" Oct 30 17:51:34.611542 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.611412 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="57ba78c3-6a27-4669-8182-35287812b3a1" containerName="payload-pre-processing" Oct 30 17:51:34.611542 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.611431 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc2fac0f-42be-41e6-8288-cdaaeaab7a1d" containerName="istio-proxy" Oct 30 17:51:34.618019 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.617995 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn"] Oct 30 17:51:34.618479 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.618109 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:34.632987 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.632958 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4a271c-openshift-default-dockercfg-46ql8\"" Oct 30 17:51:34.714951 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.713839 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf6kw\" (UniqueName: \"kubernetes.io/projected/731917aa-271d-4a54-a249-07daad43970c-kube-api-access-wf6kw\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:34.714951 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.713925 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-workload-socket\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:34.714951 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.713972 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-workload-certs\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:34.714951 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.714023 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-credential-socket\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:34.714951 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.714090 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/731917aa-271d-4a54-a249-07daad43970c-istio-token\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:34.714951 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.714138 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-istio-data\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:34.714951 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.714177 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/731917aa-271d-4a54-a249-07daad43970c-istio-podinfo\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:34.714951 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.714243 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-istio-envoy\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:34.714951 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.714330 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/731917aa-271d-4a54-a249-07daad43970c-istiod-ca-cert\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:34.815694 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.815657 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/731917aa-271d-4a54-a249-07daad43970c-istiod-ca-cert\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:34.815933 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.815750 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf6kw\" (UniqueName: \"kubernetes.io/projected/731917aa-271d-4a54-a249-07daad43970c-kube-api-access-wf6kw\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:34.815933 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.815777 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-workload-socket\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:34.815933 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.815807 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-workload-certs\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:34.815933 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.815843 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-credential-socket\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:34.815933 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.815927 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/731917aa-271d-4a54-a249-07daad43970c-istio-token\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:34.816313 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.815962 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-istio-data\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:34.816313 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.815993 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/731917aa-271d-4a54-a249-07daad43970c-istio-podinfo\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:34.816313 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.816046 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-istio-envoy\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:34.816541 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.816513 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-credential-socket\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:34.816825 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.816796 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-istio-data\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:34.816942 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.816893 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-workload-certs\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:34.817036 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.817006 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-workload-socket\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:34.817429 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.817405 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/731917aa-271d-4a54-a249-07daad43970c-istiod-ca-cert\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:34.818844 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.818827 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-istio-envoy\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:34.819018 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.818998 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/731917aa-271d-4a54-a249-07daad43970c-istio-podinfo\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:34.839552 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.839529 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/731917aa-271d-4a54-a249-07daad43970c-istio-token\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:34.839777 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.839754 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf6kw\" (UniqueName: \"kubernetes.io/projected/731917aa-271d-4a54-a249-07daad43970c-kube-api-access-wf6kw\") pod \"e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:34.936070 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:34.936038 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:35.219903 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:35.216285 4965 patch_prober.go:29] interesting pod/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 17:51:35.219903 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:35.216357 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" podUID="b8cbab33-ea35-4e2e-a82d-74e1f47c5b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 17:51:35.402039 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:35.397378 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 17:51:35.402039 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:35.397458 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" podUID="b48e4e05-e95d-424c-887e-668bdc42e398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 17:51:35.614141 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:35.614061 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn"] Oct 30 17:51:35.731506 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:51:35.731475 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod731917aa_271d_4a54_a249_07daad43970c.slice/crio-eece93b9a66ed6c5557d1b16bc68b03402b2c5a43805c00361778353c7848d0e WatchSource:0}: Error finding container eece93b9a66ed6c5557d1b16bc68b03402b2c5a43805c00361778353c7848d0e: Status 404 returned error can't find the container with id eece93b9a66ed6c5557d1b16bc68b03402b2c5a43805c00361778353c7848d0e Oct 30 17:51:35.734454 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:35.734418 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:51:35.734542 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:35.734489 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:51:35.734542 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:35.734516 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:51:36.042438 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:51:36.042347 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:51:36.222894 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:36.221971 4965 patch_prober.go:29] interesting pod/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 17:51:36.222894 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:36.222026 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" podUID="b8cbab33-ea35-4e2e-a82d-74e1f47c5b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 17:51:36.312908 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:36.312761 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:51:36.397095 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:36.397005 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 17:51:36.397095 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:36.397068 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" podUID="b48e4e05-e95d-424c-887e-668bdc42e398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 17:51:36.741071 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:36.741034 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" event={"ID":"731917aa-271d-4a54-a249-07daad43970c","Type":"ContainerStarted","Data":"a6b699b93042338bae23e1d9a1cdc11e9c67ff16bd3a9799ee5f3ef0a54bd28b"} Oct 30 17:51:36.741493 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:36.741100 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" event={"ID":"731917aa-271d-4a54-a249-07daad43970c","Type":"ContainerStarted","Data":"eece93b9a66ed6c5557d1b16bc68b03402b2c5a43805c00361778353c7848d0e"} Oct 30 17:51:36.815887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:36.815062 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" podStartSLOduration=2.815017025 podStartE2EDuration="2.815017025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:36.7873242 +0000 UTC m=+2354.839424132" watchObservedRunningTime="2026-10-30 17:51:36.815017025 +0000 UTC m=+2354.867116957" Oct 30 17:51:36.937170 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:36.937135 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:36.939251 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:36.939224 4965 patch_prober.go:29] interesting pod/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 17:51:36.939398 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:36.939271 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" podUID="731917aa-271d-4a54-a249-07daad43970c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 17:51:37.229885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:37.218273 4965 patch_prober.go:29] interesting pod/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 30 17:51:37.229885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:37.218334 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" podUID="b8cbab33-ea35-4e2e-a82d-74e1f47c5b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 30 17:51:37.390302 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:37.390268 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:51:37.391842 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:37.391807 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 17:51:37.940756 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:37.939240 4965 patch_prober.go:29] interesting pod/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 17:51:37.940756 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:37.939304 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" podUID="731917aa-271d-4a54-a249-07daad43970c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 17:51:38.221690 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:38.221612 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:51:38.222775 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:38.222751 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:51:38.936445 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:38.936413 4965 patch_prober.go:29] interesting pod/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 17:51:38.936630 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:38.936474 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" podUID="731917aa-271d-4a54-a249-07daad43970c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 17:51:39.955926 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:39.950687 4965 patch_prober.go:29] interesting pod/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 17:51:39.955926 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:39.950755 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" podUID="731917aa-271d-4a54-a249-07daad43970c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 17:51:40.936589 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:40.936551 4965 patch_prober.go:29] interesting pod/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 17:51:40.936776 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:40.936601 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" podUID="731917aa-271d-4a54-a249-07daad43970c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 17:51:41.937302 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:41.937271 4965 patch_prober.go:29] interesting pod/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 17:51:41.937701 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:41.937331 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" podUID="731917aa-271d-4a54-a249-07daad43970c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 17:51:43.065910 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:43.062920 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:43.065910 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:43.063746 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:43.065910 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:43.065218 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:51:51.155930 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:51.155896 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45"] Oct 30 17:51:51.156338 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:51.156086 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" podUID="e75131de-9643-4578-9a0a-e5c41d409131" containerName="istio-proxy" containerID="cri-o://664fc534a0f87d9f7871e86fcf1e33be226c2366598d850f3df7d36b2a3e31b5" gracePeriod=30 Oct 30 17:51:56.463995 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:56.463967 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:51:56.505257 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:56.505225 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e75131de-9643-4578-9a0a-e5c41d409131-kube-api-access-c4j4l\" (UniqueName: \"kubernetes.io/projected/e75131de-9643-4578-9a0a-e5c41d409131-kube-api-access-c4j4l\") pod \"e75131de-9643-4578-9a0a-e5c41d409131\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " Oct 30 17:51:56.505465 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:56.505278 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-istio-data\") pod \"e75131de-9643-4578-9a0a-e5c41d409131\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " Oct 30 17:51:56.505465 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:56.505321 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-workload-certs\") pod \"e75131de-9643-4578-9a0a-e5c41d409131\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " Oct 30 17:51:56.505465 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:56.505358 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e75131de-9643-4578-9a0a-e5c41d409131-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e75131de-9643-4578-9a0a-e5c41d409131-istio-podinfo\") pod \"e75131de-9643-4578-9a0a-e5c41d409131\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " Oct 30 17:51:56.505465 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:56.505415 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e75131de-9643-4578-9a0a-e5c41d409131-istio-token\" (UniqueName: \"kubernetes.io/projected/e75131de-9643-4578-9a0a-e5c41d409131-istio-token\") pod \"e75131de-9643-4578-9a0a-e5c41d409131\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " Oct 30 17:51:56.505699 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:56.505498 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e75131de-9643-4578-9a0a-e5c41d409131-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e75131de-9643-4578-9a0a-e5c41d409131-istiod-ca-cert\") pod \"e75131de-9643-4578-9a0a-e5c41d409131\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " Oct 30 17:51:56.505699 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:56.505654 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-istio-data" pod "e75131de-9643-4578-9a0a-e5c41d409131" (UID: "e75131de-9643-4578-9a0a-e5c41d409131"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:56.505699 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:56.505674 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-istio-envoy\") pod \"e75131de-9643-4578-9a0a-e5c41d409131\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " Oct 30 17:51:56.505856 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:56.505780 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-workload-socket\") pod \"e75131de-9643-4578-9a0a-e5c41d409131\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " Oct 30 17:51:56.505944 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:56.505912 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-credential-socket\") pod \"e75131de-9643-4578-9a0a-e5c41d409131\" (UID: \"e75131de-9643-4578-9a0a-e5c41d409131\") " Oct 30 17:51:56.507272 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:56.505670 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-workload-certs" pod "e75131de-9643-4578-9a0a-e5c41d409131" (UID: "e75131de-9643-4578-9a0a-e5c41d409131"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:56.507272 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:56.506435 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-credential-socket" pod "e75131de-9643-4578-9a0a-e5c41d409131" (UID: "e75131de-9643-4578-9a0a-e5c41d409131"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:56.507272 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:56.506693 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-workload-socket" pod "e75131de-9643-4578-9a0a-e5c41d409131" (UID: "e75131de-9643-4578-9a0a-e5c41d409131"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:56.507272 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:56.507121 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:51:56.507272 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:56.507145 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:51:56.507272 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:56.507160 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:51:56.507272 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:56.507174 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:51:56.508221 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:56.508184 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e75131de-9643-4578-9a0a-e5c41d409131-istiod-ca-cert" pod "e75131de-9643-4578-9a0a-e5c41d409131" (UID: "e75131de-9643-4578-9a0a-e5c41d409131"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:51:56.508508 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:56.508482 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e75131de-9643-4578-9a0a-e5c41d409131-kube-api-access-c4j4l" pod "e75131de-9643-4578-9a0a-e5c41d409131" (UID: "e75131de-9643-4578-9a0a-e5c41d409131"). InnerVolumeSpecName "kube-api-access-c4j4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:51:56.508762 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:56.508734 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-istio-envoy" pod "e75131de-9643-4578-9a0a-e5c41d409131" (UID: "e75131de-9643-4578-9a0a-e5c41d409131"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:56.508851 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:56.508824 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e75131de-9643-4578-9a0a-e5c41d409131-istio-token" pod "e75131de-9643-4578-9a0a-e5c41d409131" (UID: "e75131de-9643-4578-9a0a-e5c41d409131"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:51:56.509068 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:56.509046 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e75131de-9643-4578-9a0a-e5c41d409131-istio-podinfo" pod "e75131de-9643-4578-9a0a-e5c41d409131" (UID: "e75131de-9643-4578-9a0a-e5c41d409131"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:51:56.608447 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:56.608406 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4j4l\" (UniqueName: \"kubernetes.io/projected/e75131de-9643-4578-9a0a-e5c41d409131-kube-api-access-c4j4l\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:51:56.608447 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:56.608440 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e75131de-9643-4578-9a0a-e5c41d409131-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:51:56.608447 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:56.608452 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e75131de-9643-4578-9a0a-e5c41d409131-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:51:56.608685 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:56.608464 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e75131de-9643-4578-9a0a-e5c41d409131-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:51:56.608685 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:56.608477 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e75131de-9643-4578-9a0a-e5c41d409131-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:51:57.070411 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:57.070360 4965 generic.go:356] "Generic (PLEG): container finished" podID="e75131de-9643-4578-9a0a-e5c41d409131" containerID="664fc534a0f87d9f7871e86fcf1e33be226c2366598d850f3df7d36b2a3e31b5" exitCode=0 Oct 30 17:51:57.070411 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:57.070408 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" Oct 30 17:51:57.070650 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:57.070441 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" event={"ID":"e75131de-9643-4578-9a0a-e5c41d409131","Type":"ContainerDied","Data":"664fc534a0f87d9f7871e86fcf1e33be226c2366598d850f3df7d36b2a3e31b5"} Oct 30 17:51:57.070650 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:57.070495 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45" event={"ID":"e75131de-9643-4578-9a0a-e5c41d409131","Type":"ContainerDied","Data":"2acecfc27934ed610edb11f96cb0255a922f71d822de5b77ba21f952df19702e"} Oct 30 17:51:57.070650 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:57.070522 4965 scope.go:122] "RemoveContainer" containerID="664fc534a0f87d9f7871e86fcf1e33be226c2366598d850f3df7d36b2a3e31b5" Oct 30 17:51:57.082522 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:57.082505 4965 scope.go:122] "RemoveContainer" containerID="664fc534a0f87d9f7871e86fcf1e33be226c2366598d850f3df7d36b2a3e31b5" Oct 30 17:51:57.082821 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:51:57.082800 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"664fc534a0f87d9f7871e86fcf1e33be226c2366598d850f3df7d36b2a3e31b5\": container with ID starting with 664fc534a0f87d9f7871e86fcf1e33be226c2366598d850f3df7d36b2a3e31b5 not found: ID does not exist" containerID="664fc534a0f87d9f7871e86fcf1e33be226c2366598d850f3df7d36b2a3e31b5" Oct 30 17:51:57.082908 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:57.082831 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"664fc534a0f87d9f7871e86fcf1e33be226c2366598d850f3df7d36b2a3e31b5"} err="failed to get container status \"664fc534a0f87d9f7871e86fcf1e33be226c2366598d850f3df7d36b2a3e31b5\": rpc error: code = NotFound desc = could not find container \"664fc534a0f87d9f7871e86fcf1e33be226c2366598d850f3df7d36b2a3e31b5\": container with ID starting with 664fc534a0f87d9f7871e86fcf1e33be226c2366598d850f3df7d36b2a3e31b5 not found: ID does not exist" Oct 30 17:51:57.124304 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:57.124274 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45"] Oct 30 17:51:57.136696 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:57.136664 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-4wx45"] Oct 30 17:51:58.102702 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:51:58.102662 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e75131de-9643-4578-9a0a-e5c41d409131" path="/var/lib/kubelet/pods/e75131de-9643-4578-9a0a-e5c41d409131/volumes" Oct 30 17:52:02.235169 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:02.235135 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b1c345-d4898585f-6kvs7"] Oct 30 17:52:03.443174 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:03.443090 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:52:06.024488 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:52:06.024443 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:52:07.235598 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:07.235545 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1c345-d4898585f-6kvs7" podUID="e0d95f31-b1ea-4a3c-92a3-1118993629ed" containerName="payload-pre-processing" containerID="cri-o://a79ea137cf3be37bd33540a40e65140f4dffef8add43a2a423b33a0946e15b5b" gracePeriod=25 Oct 30 17:52:12.483263 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:12.483220 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1c345-d4898585f-6kvs7" podUID="e0d95f31-b1ea-4a3c-92a3-1118993629ed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:52:22.532886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:22.516034 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1c345-d4898585f-6kvs7" podUID="e0d95f31-b1ea-4a3c-92a3-1118993629ed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:52:22.609785 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:22.609758 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 17:52:22.625322 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:22.625296 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 17:52:24.654625 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:24.654589 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g"] Oct 30 17:52:24.655414 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:24.655396 4965 cpu_manager.go:414] "containerMap: removing container" podUID="e75131de-9643-4578-9a0a-e5c41d409131" containerName="istio-proxy" Oct 30 17:52:24.655414 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:24.655412 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e75131de-9643-4578-9a0a-e5c41d409131" containerName="istio-proxy" Oct 30 17:52:24.655598 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:24.655525 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="e75131de-9643-4578-9a0a-e5c41d409131" containerName="istio-proxy" Oct 30 17:52:24.659463 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:24.659443 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g" Oct 30 17:52:24.686692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:24.686667 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-a2a6cb-serving-cert\"" Oct 30 17:52:24.693198 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:24.693171 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g"] Oct 30 17:52:24.770225 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:24.770193 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40e1abc1-ddb7-44f3-b112-e704e1ebbc81-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g\" (UID: \"40e1abc1-ddb7-44f3-b112-e704e1ebbc81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g" Oct 30 17:52:24.770225 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:24.770237 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8fpg\" (UniqueName: \"kubernetes.io/projected/40e1abc1-ddb7-44f3-b112-e704e1ebbc81-kube-api-access-k8fpg\") pod \"maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g\" (UID: \"40e1abc1-ddb7-44f3-b112-e704e1ebbc81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g" Oct 30 17:52:24.770522 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:24.770496 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40e1abc1-ddb7-44f3-b112-e704e1ebbc81-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g\" (UID: \"40e1abc1-ddb7-44f3-b112-e704e1ebbc81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g" Oct 30 17:52:24.770629 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:24.770607 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40e1abc1-ddb7-44f3-b112-e704e1ebbc81-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g\" (UID: \"40e1abc1-ddb7-44f3-b112-e704e1ebbc81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g" Oct 30 17:52:24.875908 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:24.875626 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40e1abc1-ddb7-44f3-b112-e704e1ebbc81-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g\" (UID: \"40e1abc1-ddb7-44f3-b112-e704e1ebbc81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g" Oct 30 17:52:24.875908 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:24.875751 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40e1abc1-ddb7-44f3-b112-e704e1ebbc81-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g\" (UID: \"40e1abc1-ddb7-44f3-b112-e704e1ebbc81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g" Oct 30 17:52:24.875908 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:24.875817 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40e1abc1-ddb7-44f3-b112-e704e1ebbc81-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g\" (UID: \"40e1abc1-ddb7-44f3-b112-e704e1ebbc81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g" Oct 30 17:52:24.875908 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:24.875878 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8fpg\" (UniqueName: \"kubernetes.io/projected/40e1abc1-ddb7-44f3-b112-e704e1ebbc81-kube-api-access-k8fpg\") pod \"maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g\" (UID: \"40e1abc1-ddb7-44f3-b112-e704e1ebbc81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g" Oct 30 17:52:24.883962 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:24.883926 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40e1abc1-ddb7-44f3-b112-e704e1ebbc81-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g\" (UID: \"40e1abc1-ddb7-44f3-b112-e704e1ebbc81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g" Oct 30 17:52:24.890882 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:24.890790 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40e1abc1-ddb7-44f3-b112-e704e1ebbc81-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g\" (UID: \"40e1abc1-ddb7-44f3-b112-e704e1ebbc81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g" Oct 30 17:52:24.895923 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:24.895893 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40e1abc1-ddb7-44f3-b112-e704e1ebbc81-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g\" (UID: \"40e1abc1-ddb7-44f3-b112-e704e1ebbc81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g" Oct 30 17:52:24.920337 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:24.920261 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8fpg\" (UniqueName: \"kubernetes.io/projected/40e1abc1-ddb7-44f3-b112-e704e1ebbc81-kube-api-access-k8fpg\") pod \"maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g\" (UID: \"40e1abc1-ddb7-44f3-b112-e704e1ebbc81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g" Oct 30 17:52:24.973818 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:24.973784 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g" Oct 30 17:52:25.287782 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:25.287754 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g"] Oct 30 17:52:25.287913 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:52:25.287758 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e1abc1_ddb7_44f3_b112_e704e1ebbc81.slice/crio-6cec52d3a932f86569779fcc2b64486b6944432532b7de15651ad47e803779c4 WatchSource:0}: Error finding container 6cec52d3a932f86569779fcc2b64486b6944432532b7de15651ad47e803779c4: Status 404 returned error can't find the container with id 6cec52d3a932f86569779fcc2b64486b6944432532b7de15651ad47e803779c4 Oct 30 17:52:25.516179 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:25.516140 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g" event={"ID":"40e1abc1-ddb7-44f3-b112-e704e1ebbc81","Type":"ContainerStarted","Data":"6cec52d3a932f86569779fcc2b64486b6944432532b7de15651ad47e803779c4"} Oct 30 17:52:26.527133 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:26.527089 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g" event={"ID":"40e1abc1-ddb7-44f3-b112-e704e1ebbc81","Type":"ContainerStarted","Data":"e413056d747329120e70b395e491aefa0f47dd5f131ce930c01e76078739f8b4"} Oct 30 17:52:26.527776 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:26.527752 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g" Oct 30 17:52:26.575239 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:26.575191 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g" podStartSLOduration=2.251580871 podStartE2EDuration="2.575176293s" totalImagesPullingTime="323.595422ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:52:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:26.572286605 +0000 UTC m=+2404.624386536" watchObservedRunningTime="2026-10-30 17:52:26.575176293 +0000 UTC m=+2404.627276244" Oct 30 17:52:27.536623 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:27.536589 4965 generic.go:356] "Generic (PLEG): container finished" podID="e0d95f31-b1ea-4a3c-92a3-1118993629ed" containerID="a79ea137cf3be37bd33540a40e65140f4dffef8add43a2a423b33a0946e15b5b" exitCode=0 Oct 30 17:52:27.537017 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:27.536671 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1c345-d4898585f-6kvs7" event={"ID":"e0d95f31-b1ea-4a3c-92a3-1118993629ed","Type":"ContainerDied","Data":"a79ea137cf3be37bd33540a40e65140f4dffef8add43a2a423b33a0946e15b5b"} Oct 30 17:52:27.537017 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:27.536738 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1c345-d4898585f-6kvs7" event={"ID":"e0d95f31-b1ea-4a3c-92a3-1118993629ed","Type":"ContainerDied","Data":"82bf539e1af0ca58e9b36a8f00f8b17af22bd33358ce67abb8635b9dd82ba0f6"} Oct 30 17:52:27.537017 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:27.536770 4965 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82bf539e1af0ca58e9b36a8f00f8b17af22bd33358ce67abb8635b9dd82ba0f6" Oct 30 17:52:27.553923 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:27.553896 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1c345-d4898585f-6kvs7" Oct 30 17:52:27.644633 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:27.644603 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0d95f31-b1ea-4a3c-92a3-1118993629ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d95f31-b1ea-4a3c-92a3-1118993629ed-plugins-config-volume\") pod \"e0d95f31-b1ea-4a3c-92a3-1118993629ed\" (UID: \"e0d95f31-b1ea-4a3c-92a3-1118993629ed\") " Oct 30 17:52:27.644829 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:27.644809 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0d95f31-b1ea-4a3c-92a3-1118993629ed-kube-api-access-mvjhz\" (UniqueName: \"kubernetes.io/projected/e0d95f31-b1ea-4a3c-92a3-1118993629ed-kube-api-access-mvjhz\") pod \"e0d95f31-b1ea-4a3c-92a3-1118993629ed\" (UID: \"e0d95f31-b1ea-4a3c-92a3-1118993629ed\") " Oct 30 17:52:27.645076 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:27.645052 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d95f31-b1ea-4a3c-92a3-1118993629ed-plugins-config-volume" pod "e0d95f31-b1ea-4a3c-92a3-1118993629ed" (UID: "e0d95f31-b1ea-4a3c-92a3-1118993629ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:52:27.645561 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:27.645534 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d95f31-b1ea-4a3c-92a3-1118993629ed-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:52:27.647052 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:27.647022 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d95f31-b1ea-4a3c-92a3-1118993629ed-kube-api-access-mvjhz" pod "e0d95f31-b1ea-4a3c-92a3-1118993629ed" (UID: "e0d95f31-b1ea-4a3c-92a3-1118993629ed"). InnerVolumeSpecName "kube-api-access-mvjhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:52:27.747300 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:27.747269 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvjhz\" (UniqueName: \"kubernetes.io/projected/e0d95f31-b1ea-4a3c-92a3-1118993629ed-kube-api-access-mvjhz\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:52:28.588886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:28.585511 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1c345-d4898585f-6kvs7" Oct 30 17:52:28.727185 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:28.727152 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b1c345-d4898585f-6kvs7"] Oct 30 17:52:28.731453 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:28.731224 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b1c345-d4898585f-6kvs7"] Oct 30 17:52:30.103322 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:30.103287 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0d95f31-b1ea-4a3c-92a3-1118993629ed" path="/var/lib/kubelet/pods/e0d95f31-b1ea-4a3c-92a3-1118993629ed/volumes" Oct 30 17:52:32.542708 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:32.542679 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g" Oct 30 17:52:36.023753 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:52:36.023706 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:52:41.319985 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:41.319954 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-303ba5-7bfc9db6c8-shn24"] Oct 30 17:52:46.320396 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:46.320352 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-303ba5-7bfc9db6c8-shn24" podUID="0120bd60-ccb9-4b8d-99ec-04a9fba855d2" containerName="payload-processing" containerID="cri-o://024fecdf02b80a7328b7f079b08f6ff16445fe37496baa454949d7e6ba1087c2" gracePeriod=25 Oct 30 17:52:46.617831 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:46.617801 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-303ba5-7bfc9db6c8-shn24" Oct 30 17:52:46.702772 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:46.702741 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0120bd60-ccb9-4b8d-99ec-04a9fba855d2-kube-api-access-jkb8n\" (UniqueName: \"kubernetes.io/projected/0120bd60-ccb9-4b8d-99ec-04a9fba855d2-kube-api-access-jkb8n\") pod \"0120bd60-ccb9-4b8d-99ec-04a9fba855d2\" (UID: \"0120bd60-ccb9-4b8d-99ec-04a9fba855d2\") " Oct 30 17:52:46.703007 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:46.702828 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0120bd60-ccb9-4b8d-99ec-04a9fba855d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0120bd60-ccb9-4b8d-99ec-04a9fba855d2-plugins-config-volume\") pod \"0120bd60-ccb9-4b8d-99ec-04a9fba855d2\" (UID: \"0120bd60-ccb9-4b8d-99ec-04a9fba855d2\") " Oct 30 17:52:46.703281 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:46.703254 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0120bd60-ccb9-4b8d-99ec-04a9fba855d2-plugins-config-volume" pod "0120bd60-ccb9-4b8d-99ec-04a9fba855d2" (UID: "0120bd60-ccb9-4b8d-99ec-04a9fba855d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:52:46.703585 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:46.703565 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0120bd60-ccb9-4b8d-99ec-04a9fba855d2-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:52:46.705137 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:46.705111 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0120bd60-ccb9-4b8d-99ec-04a9fba855d2-kube-api-access-jkb8n" pod "0120bd60-ccb9-4b8d-99ec-04a9fba855d2" (UID: "0120bd60-ccb9-4b8d-99ec-04a9fba855d2"). InnerVolumeSpecName "kube-api-access-jkb8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:52:46.766038 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:46.766004 4965 generic.go:356] "Generic (PLEG): container finished" podID="0120bd60-ccb9-4b8d-99ec-04a9fba855d2" containerID="024fecdf02b80a7328b7f079b08f6ff16445fe37496baa454949d7e6ba1087c2" exitCode=0 Oct 30 17:52:46.766201 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:46.766054 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-303ba5-7bfc9db6c8-shn24" Oct 30 17:52:46.766201 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:46.766088 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-303ba5-7bfc9db6c8-shn24" event={"ID":"0120bd60-ccb9-4b8d-99ec-04a9fba855d2","Type":"ContainerDied","Data":"024fecdf02b80a7328b7f079b08f6ff16445fe37496baa454949d7e6ba1087c2"} Oct 30 17:52:46.766201 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:46.766144 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-303ba5-7bfc9db6c8-shn24" event={"ID":"0120bd60-ccb9-4b8d-99ec-04a9fba855d2","Type":"ContainerDied","Data":"9856139f64755dacdf475af4fd094b31a97d8d79ebfe22a532e6e9d127235f3b"} Oct 30 17:52:46.766201 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:46.766165 4965 scope.go:122] "RemoveContainer" containerID="024fecdf02b80a7328b7f079b08f6ff16445fe37496baa454949d7e6ba1087c2" Oct 30 17:52:46.777696 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:46.777671 4965 scope.go:122] "RemoveContainer" containerID="024fecdf02b80a7328b7f079b08f6ff16445fe37496baa454949d7e6ba1087c2" Oct 30 17:52:46.778027 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:52:46.778005 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"024fecdf02b80a7328b7f079b08f6ff16445fe37496baa454949d7e6ba1087c2\": container with ID starting with 024fecdf02b80a7328b7f079b08f6ff16445fe37496baa454949d7e6ba1087c2 not found: ID does not exist" containerID="024fecdf02b80a7328b7f079b08f6ff16445fe37496baa454949d7e6ba1087c2" Oct 30 17:52:46.778111 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:46.778039 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"024fecdf02b80a7328b7f079b08f6ff16445fe37496baa454949d7e6ba1087c2"} err="failed to get container status \"024fecdf02b80a7328b7f079b08f6ff16445fe37496baa454949d7e6ba1087c2\": rpc error: code = NotFound desc = could not find container \"024fecdf02b80a7328b7f079b08f6ff16445fe37496baa454949d7e6ba1087c2\": container with ID starting with 024fecdf02b80a7328b7f079b08f6ff16445fe37496baa454949d7e6ba1087c2 not found: ID does not exist" Oct 30 17:52:46.862316 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:46.862214 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkb8n\" (UniqueName: \"kubernetes.io/projected/0120bd60-ccb9-4b8d-99ec-04a9fba855d2-kube-api-access-jkb8n\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:52:46.888942 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:46.888909 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-303ba5-7bfc9db6c8-shn24"] Oct 30 17:52:46.923890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:46.920312 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-303ba5-7bfc9db6c8-shn24"] Oct 30 17:52:48.101017 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:48.100983 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0120bd60-ccb9-4b8d-99ec-04a9fba855d2" path="/var/lib/kubelet/pods/0120bd60-ccb9-4b8d-99ec-04a9fba855d2/volumes" Oct 30 17:52:49.410890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:52:49.408234 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:53:06.000277 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:53:06.000236 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:53:14.510751 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:14.510704 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx"] Oct 30 17:53:14.511516 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:14.510952 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx" podUID="ee3c3584-46f0-432e-b703-702207a1bca0" containerName="maas-api" containerID="cri-o://64d8d8e72eeddfcef6fab23f72321aaa48e2b6fac464d17b0ea57584afeee88d" gracePeriod=30 Oct 30 17:53:14.897045 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:14.894934 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx" Oct 30 17:53:15.029884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:15.029361 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee3c3584-46f0-432e-b703-702207a1bca0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee3c3584-46f0-432e-b703-702207a1bca0-openshift-service-ca\") pod \"ee3c3584-46f0-432e-b703-702207a1bca0\" (UID: \"ee3c3584-46f0-432e-b703-702207a1bca0\") " Oct 30 17:53:15.029884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:15.029449 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee3c3584-46f0-432e-b703-702207a1bca0-kube-api-access-9pb4b\" (UniqueName: \"kubernetes.io/projected/ee3c3584-46f0-432e-b703-702207a1bca0-kube-api-access-9pb4b\") pod \"ee3c3584-46f0-432e-b703-702207a1bca0\" (UID: \"ee3c3584-46f0-432e-b703-702207a1bca0\") " Oct 30 17:53:15.029884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:15.029479 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee3c3584-46f0-432e-b703-702207a1bca0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee3c3584-46f0-432e-b703-702207a1bca0-maas-api-tls\") pod \"ee3c3584-46f0-432e-b703-702207a1bca0\" (UID: \"ee3c3584-46f0-432e-b703-702207a1bca0\") " Oct 30 17:53:15.029884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:15.029819 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee3c3584-46f0-432e-b703-702207a1bca0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee3c3584-46f0-432e-b703-702207a1bca0-metrics-certs\") pod \"ee3c3584-46f0-432e-b703-702207a1bca0\" (UID: \"ee3c3584-46f0-432e-b703-702207a1bca0\") " Oct 30 17:53:15.041033 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:15.034643 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee3c3584-46f0-432e-b703-702207a1bca0-metrics-certs" pod "ee3c3584-46f0-432e-b703-702207a1bca0" (UID: "ee3c3584-46f0-432e-b703-702207a1bca0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:15.041033 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:15.035054 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee3c3584-46f0-432e-b703-702207a1bca0-openshift-service-ca" pod "ee3c3584-46f0-432e-b703-702207a1bca0" (UID: "ee3c3584-46f0-432e-b703-702207a1bca0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:53:15.148935 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:15.148831 4965 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee3c3584-46f0-432e-b703-702207a1bca0-openshift-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:53:15.148935 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:15.148888 4965 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee3c3584-46f0-432e-b703-702207a1bca0-metrics-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:53:15.169895 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:15.166984 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee3c3584-46f0-432e-b703-702207a1bca0-maas-api-tls" pod "ee3c3584-46f0-432e-b703-702207a1bca0" (UID: "ee3c3584-46f0-432e-b703-702207a1bca0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:15.174967 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:15.174937 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3c3584-46f0-432e-b703-702207a1bca0-kube-api-access-9pb4b" pod "ee3c3584-46f0-432e-b703-702207a1bca0" (UID: "ee3c3584-46f0-432e-b703-702207a1bca0"). InnerVolumeSpecName "kube-api-access-9pb4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:53:15.202947 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:15.202909 4965 generic.go:356] "Generic (PLEG): container finished" podID="ee3c3584-46f0-432e-b703-702207a1bca0" containerID="64d8d8e72eeddfcef6fab23f72321aaa48e2b6fac464d17b0ea57584afeee88d" exitCode=0 Oct 30 17:53:15.203139 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:15.203022 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx" event={"ID":"ee3c3584-46f0-432e-b703-702207a1bca0","Type":"ContainerDied","Data":"64d8d8e72eeddfcef6fab23f72321aaa48e2b6fac464d17b0ea57584afeee88d"} Oct 30 17:53:15.203139 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:15.203075 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx" event={"ID":"ee3c3584-46f0-432e-b703-702207a1bca0","Type":"ContainerDied","Data":"aaf8048580fe7bd26daad1c2c1b9928aa414dbf1dd6d442723d46bf86e6f29e8"} Oct 30 17:53:15.203139 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:15.203104 4965 scope.go:122] "RemoveContainer" containerID="64d8d8e72eeddfcef6fab23f72321aaa48e2b6fac464d17b0ea57584afeee88d" Oct 30 17:53:15.203326 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:15.203267 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx" Oct 30 17:53:15.239024 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:15.238997 4965 scope.go:122] "RemoveContainer" containerID="64d8d8e72eeddfcef6fab23f72321aaa48e2b6fac464d17b0ea57584afeee88d" Oct 30 17:53:15.239383 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:53:15.239355 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64d8d8e72eeddfcef6fab23f72321aaa48e2b6fac464d17b0ea57584afeee88d\": container with ID starting with 64d8d8e72eeddfcef6fab23f72321aaa48e2b6fac464d17b0ea57584afeee88d not found: ID does not exist" containerID="64d8d8e72eeddfcef6fab23f72321aaa48e2b6fac464d17b0ea57584afeee88d" Oct 30 17:53:15.239514 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:15.239391 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64d8d8e72eeddfcef6fab23f72321aaa48e2b6fac464d17b0ea57584afeee88d"} err="failed to get container status \"64d8d8e72eeddfcef6fab23f72321aaa48e2b6fac464d17b0ea57584afeee88d\": rpc error: code = NotFound desc = could not find container \"64d8d8e72eeddfcef6fab23f72321aaa48e2b6fac464d17b0ea57584afeee88d\": container with ID starting with 64d8d8e72eeddfcef6fab23f72321aaa48e2b6fac464d17b0ea57584afeee88d not found: ID does not exist" Oct 30 17:53:15.250018 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:15.249996 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pb4b\" (UniqueName: \"kubernetes.io/projected/ee3c3584-46f0-432e-b703-702207a1bca0-kube-api-access-9pb4b\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:53:15.250145 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:15.250022 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee3c3584-46f0-432e-b703-702207a1bca0-maas-api-tls\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:53:15.268066 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:15.268044 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx"] Oct 30 17:53:15.280064 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:15.280038 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-303ba5-9dbbd8f9d-jc4fx"] Oct 30 17:53:16.100127 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:16.100085 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee3c3584-46f0-432e-b703-702207a1bca0" path="/var/lib/kubelet/pods/ee3c3584-46f0-432e-b703-702207a1bca0/volumes" Oct 30 17:53:17.467564 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:17.467528 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a23c98-787c476954-j5ls2"] Oct 30 17:53:17.468281 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:17.468266 4965 cpu_manager.go:414] "containerMap: removing container" podUID="0120bd60-ccb9-4b8d-99ec-04a9fba855d2" containerName="payload-processing" Oct 30 17:53:17.468334 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:17.468281 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0120bd60-ccb9-4b8d-99ec-04a9fba855d2" containerName="payload-processing" Oct 30 17:53:17.468334 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:17.468327 4965 cpu_manager.go:414] "containerMap: removing container" podUID="e0d95f31-b1ea-4a3c-92a3-1118993629ed" containerName="payload-pre-processing" Oct 30 17:53:17.468402 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:17.468335 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0d95f31-b1ea-4a3c-92a3-1118993629ed" containerName="payload-pre-processing" Oct 30 17:53:17.468402 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:17.468352 4965 cpu_manager.go:414] "containerMap: removing container" podUID="ee3c3584-46f0-432e-b703-702207a1bca0" containerName="maas-api" Oct 30 17:53:17.468402 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:17.468358 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee3c3584-46f0-432e-b703-702207a1bca0" containerName="maas-api" Oct 30 17:53:17.468494 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:17.468460 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0d95f31-b1ea-4a3c-92a3-1118993629ed" containerName="payload-pre-processing" Oct 30 17:53:17.468494 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:17.468471 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee3c3584-46f0-432e-b703-702207a1bca0" containerName="maas-api" Oct 30 17:53:17.468494 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:17.468478 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="0120bd60-ccb9-4b8d-99ec-04a9fba855d2" containerName="payload-processing" Oct 30 17:53:17.473587 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:17.473569 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a23c98-787c476954-j5ls2" Oct 30 17:53:17.483655 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:17.483626 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrlqw\" (UniqueName: \"kubernetes.io/projected/b6b5b4db-cdc6-4d3d-863e-678a337f87bb-kube-api-access-qrlqw\") pod \"payload-processing-e2e-shared-b-w4-a23c98-787c476954-j5ls2\" (UID: \"b6b5b4db-cdc6-4d3d-863e-678a337f87bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a23c98-787c476954-j5ls2" Oct 30 17:53:17.483871 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:17.483841 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b5b4db-cdc6-4d3d-863e-678a337f87bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-a23c98-787c476954-j5ls2\" (UID: \"b6b5b4db-cdc6-4d3d-863e-678a337f87bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a23c98-787c476954-j5ls2" Oct 30 17:53:17.491622 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:17.491597 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a23c98-787c476954-j5ls2"] Oct 30 17:53:17.497218 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:17.497198 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-a23c98-dockercfg-np4dc\"" Oct 30 17:53:17.533852 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:17.533824 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a23c98-76b6d5c87-8tx7q"] Oct 30 17:53:17.538559 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:17.538538 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a23c98-76b6d5c87-8tx7q" Oct 30 17:53:17.539006 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:17.538985 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-a23c98\"" Oct 30 17:53:17.555918 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:17.555892 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a23c98-76b6d5c87-8tx7q"] Oct 30 17:53:17.690913 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:17.688980 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b5b4db-cdc6-4d3d-863e-678a337f87bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-a23c98-787c476954-j5ls2\" (UID: \"b6b5b4db-cdc6-4d3d-863e-678a337f87bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a23c98-787c476954-j5ls2" Oct 30 17:53:17.690913 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:17.689220 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrlqw\" (UniqueName: \"kubernetes.io/projected/b6b5b4db-cdc6-4d3d-863e-678a337f87bb-kube-api-access-qrlqw\") pod \"payload-processing-e2e-shared-b-w4-a23c98-787c476954-j5ls2\" (UID: \"b6b5b4db-cdc6-4d3d-863e-678a337f87bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a23c98-787c476954-j5ls2" Oct 30 17:53:17.690913 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:17.690569 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b5b4db-cdc6-4d3d-863e-678a337f87bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-a23c98-787c476954-j5ls2\" (UID: \"b6b5b4db-cdc6-4d3d-863e-678a337f87bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a23c98-787c476954-j5ls2" Oct 30 17:53:17.790637 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:17.790500 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj957\" (UniqueName: \"kubernetes.io/projected/dea747ba-4708-4307-8744-911dea32712e-kube-api-access-rj957\") pod \"payload-pre-processing-e2e-shared-b-w4-a23c98-76b6d5c87-8tx7q\" (UID: \"dea747ba-4708-4307-8744-911dea32712e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a23c98-76b6d5c87-8tx7q" Oct 30 17:53:17.791126 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:17.791089 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dea747ba-4708-4307-8744-911dea32712e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-a23c98-76b6d5c87-8tx7q\" (UID: \"dea747ba-4708-4307-8744-911dea32712e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a23c98-76b6d5c87-8tx7q" Oct 30 17:53:17.850307 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:17.849724 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrlqw\" (UniqueName: \"kubernetes.io/projected/b6b5b4db-cdc6-4d3d-863e-678a337f87bb-kube-api-access-qrlqw\") pod \"payload-processing-e2e-shared-b-w4-a23c98-787c476954-j5ls2\" (UID: \"b6b5b4db-cdc6-4d3d-863e-678a337f87bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a23c98-787c476954-j5ls2" Oct 30 17:53:17.893117 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:17.893079 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj957\" (UniqueName: \"kubernetes.io/projected/dea747ba-4708-4307-8744-911dea32712e-kube-api-access-rj957\") pod \"payload-pre-processing-e2e-shared-b-w4-a23c98-76b6d5c87-8tx7q\" (UID: \"dea747ba-4708-4307-8744-911dea32712e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a23c98-76b6d5c87-8tx7q" Oct 30 17:53:17.893315 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:17.893131 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dea747ba-4708-4307-8744-911dea32712e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-a23c98-76b6d5c87-8tx7q\" (UID: \"dea747ba-4708-4307-8744-911dea32712e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a23c98-76b6d5c87-8tx7q" Oct 30 17:53:17.893849 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:17.893829 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dea747ba-4708-4307-8744-911dea32712e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-a23c98-76b6d5c87-8tx7q\" (UID: \"dea747ba-4708-4307-8744-911dea32712e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a23c98-76b6d5c87-8tx7q" Oct 30 17:53:17.918396 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:17.918364 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj957\" (UniqueName: \"kubernetes.io/projected/dea747ba-4708-4307-8744-911dea32712e-kube-api-access-rj957\") pod \"payload-pre-processing-e2e-shared-b-w4-a23c98-76b6d5c87-8tx7q\" (UID: \"dea747ba-4708-4307-8744-911dea32712e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a23c98-76b6d5c87-8tx7q" Oct 30 17:53:18.090672 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:18.090626 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a23c98-787c476954-j5ls2" Oct 30 17:53:18.155396 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:18.155366 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a23c98-76b6d5c87-8tx7q" Oct 30 17:53:18.397831 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:53:18.397796 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b5b4db_cdc6_4d3d_863e_678a337f87bb.slice/crio-1fd12dc8029944ae02765e9c75e5993e1f458c715abf97a0076d677328dc1eb2 WatchSource:0}: Error finding container 1fd12dc8029944ae02765e9c75e5993e1f458c715abf97a0076d677328dc1eb2: Status 404 returned error can't find the container with id 1fd12dc8029944ae02765e9c75e5993e1f458c715abf97a0076d677328dc1eb2 Oct 30 17:53:18.398706 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:18.398684 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a23c98-787c476954-j5ls2"] Oct 30 17:53:18.481476 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:18.481448 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a23c98-76b6d5c87-8tx7q"] Oct 30 17:53:18.481781 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:53:18.481469 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddea747ba_4708_4307_8744_911dea32712e.slice/crio-f680b65939013c1c412d400f8f19d443ecf078b5a38fc8a12af0598f4e328bf2 WatchSource:0}: Error finding container f680b65939013c1c412d400f8f19d443ecf078b5a38fc8a12af0598f4e328bf2: Status 404 returned error can't find the container with id f680b65939013c1c412d400f8f19d443ecf078b5a38fc8a12af0598f4e328bf2 Oct 30 17:53:19.246068 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:19.246029 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a23c98-787c476954-j5ls2" event={"ID":"b6b5b4db-cdc6-4d3d-863e-678a337f87bb","Type":"ContainerStarted","Data":"9461c21293a94982a23cbb1c06ecaf6cba2ed5b2a778ab5b3c646594d43eeb5e"} Oct 30 17:53:19.246281 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:19.246233 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a23c98-787c476954-j5ls2" Oct 30 17:53:19.246338 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:19.246285 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a23c98-787c476954-j5ls2" event={"ID":"b6b5b4db-cdc6-4d3d-863e-678a337f87bb","Type":"ContainerStarted","Data":"1fd12dc8029944ae02765e9c75e5993e1f458c715abf97a0076d677328dc1eb2"} Oct 30 17:53:19.247709 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:19.247686 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a23c98-76b6d5c87-8tx7q" event={"ID":"dea747ba-4708-4307-8744-911dea32712e","Type":"ContainerStarted","Data":"3269f404e9274f8c954539743ffe790cdeb631032e21c1a94f3cb5ca6e0d02c9"} Oct 30 17:53:19.247833 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:19.247735 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a23c98-76b6d5c87-8tx7q" event={"ID":"dea747ba-4708-4307-8744-911dea32712e","Type":"ContainerStarted","Data":"f680b65939013c1c412d400f8f19d443ecf078b5a38fc8a12af0598f4e328bf2"} Oct 30 17:53:19.247833 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:19.247762 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a23c98-76b6d5c87-8tx7q" Oct 30 17:53:19.303115 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:19.303063 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a23c98-787c476954-j5ls2" podStartSLOduration=2.303050854 podStartE2EDuration="2.303050854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:53:19.300203782 +0000 UTC m=+2457.352303718" watchObservedRunningTime="2026-10-30 17:53:19.303050854 +0000 UTC m=+2457.355150785" Oct 30 17:53:19.341834 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:19.341785 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a23c98-76b6d5c87-8tx7q" podStartSLOduration=2.341772434 podStartE2EDuration="2.341772434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:53:19.339826748 +0000 UTC m=+2457.391926698" watchObservedRunningTime="2026-10-30 17:53:19.341772434 +0000 UTC m=+2457.393872364" Oct 30 17:53:26.090040 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:26.090010 4965 scope.go:122] "RemoveContainer" containerID="a79ea137cf3be37bd33540a40e65140f4dffef8add43a2a423b33a0946e15b5b" Oct 30 17:53:30.263401 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:30.263371 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a23c98-76b6d5c87-8tx7q" Oct 30 17:53:30.263841 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:30.263466 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a23c98-787c476954-j5ls2" Oct 30 17:53:32.716993 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:32.714476 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2"] Oct 30 17:53:32.716993 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:32.714773 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" podUID="6c560829-549a-4f74-94c6-361c0662716b" containerName="istio-proxy" containerID="cri-o://3d46b129c731f4188dff5526f5df3deeb65137001d43d7f55494821423719b9a" gracePeriod=30 Oct 30 17:53:35.998479 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:53:35.998435 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:53:36.829884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:36.829346 4965 patch_prober.go:29] interesting pod/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:53:36.829884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:36.829448 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" podUID="6c560829-549a-4f74-94c6-361c0662716b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:53:38.010193 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.010168 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:53:38.054180 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.054150 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c560829-549a-4f74-94c6-361c0662716b-kube-api-access-hrqwl\" (UniqueName: \"kubernetes.io/projected/6c560829-549a-4f74-94c6-361c0662716b-kube-api-access-hrqwl\") pod \"6c560829-549a-4f74-94c6-361c0662716b\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " Oct 30 17:53:38.054374 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.054237 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c560829-549a-4f74-94c6-361c0662716b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c560829-549a-4f74-94c6-361c0662716b-istio-podinfo\") pod \"6c560829-549a-4f74-94c6-361c0662716b\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " Oct 30 17:53:38.054374 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.054353 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c560829-549a-4f74-94c6-361c0662716b-istio-token\" (UniqueName: \"kubernetes.io/projected/6c560829-549a-4f74-94c6-361c0662716b-istio-token\") pod \"6c560829-549a-4f74-94c6-361c0662716b\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " Oct 30 17:53:38.054527 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.054392 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-istio-envoy\") pod \"6c560829-549a-4f74-94c6-361c0662716b\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " Oct 30 17:53:38.054527 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.054471 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-workload-certs\") pod \"6c560829-549a-4f74-94c6-361c0662716b\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " Oct 30 17:53:38.054646 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.054534 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-workload-socket\") pod \"6c560829-549a-4f74-94c6-361c0662716b\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " Oct 30 17:53:38.054715 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.054674 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-istio-data\") pod \"6c560829-549a-4f74-94c6-361c0662716b\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " Oct 30 17:53:38.054776 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.054738 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-credential-socket\") pod \"6c560829-549a-4f74-94c6-361c0662716b\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " Oct 30 17:53:38.054828 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.054804 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c560829-549a-4f74-94c6-361c0662716b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c560829-549a-4f74-94c6-361c0662716b-istiod-ca-cert\") pod \"6c560829-549a-4f74-94c6-361c0662716b\" (UID: \"6c560829-549a-4f74-94c6-361c0662716b\") " Oct 30 17:53:38.054926 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.054901 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-workload-certs" pod "6c560829-549a-4f74-94c6-361c0662716b" (UID: "6c560829-549a-4f74-94c6-361c0662716b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:53:38.055032 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.055007 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-istio-data" pod "6c560829-549a-4f74-94c6-361c0662716b" (UID: "6c560829-549a-4f74-94c6-361c0662716b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:53:38.055131 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.055106 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-credential-socket" pod "6c560829-549a-4f74-94c6-361c0662716b" (UID: "6c560829-549a-4f74-94c6-361c0662716b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:53:38.055190 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.055136 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-workload-socket" pod "6c560829-549a-4f74-94c6-361c0662716b" (UID: "6c560829-549a-4f74-94c6-361c0662716b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:53:38.055383 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.055356 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c560829-549a-4f74-94c6-361c0662716b-istiod-ca-cert" pod "6c560829-549a-4f74-94c6-361c0662716b" (UID: "6c560829-549a-4f74-94c6-361c0662716b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:53:38.056029 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.056003 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:53:38.056131 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.056033 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:53:38.056131 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.056048 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c560829-549a-4f74-94c6-361c0662716b-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:53:38.056131 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.056063 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:53:38.056131 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.056076 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:53:38.057923 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.057854 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c560829-549a-4f74-94c6-361c0662716b-istio-token" pod "6c560829-549a-4f74-94c6-361c0662716b" (UID: "6c560829-549a-4f74-94c6-361c0662716b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:53:38.057923 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.057919 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c560829-549a-4f74-94c6-361c0662716b-kube-api-access-hrqwl" pod "6c560829-549a-4f74-94c6-361c0662716b" (UID: "6c560829-549a-4f74-94c6-361c0662716b"). InnerVolumeSpecName "kube-api-access-hrqwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:53:38.065279 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.065244 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-istio-envoy" pod "6c560829-549a-4f74-94c6-361c0662716b" (UID: "6c560829-549a-4f74-94c6-361c0662716b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:53:38.096261 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.090986 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c560829-549a-4f74-94c6-361c0662716b-istio-podinfo" pod "6c560829-549a-4f74-94c6-361c0662716b" (UID: "6c560829-549a-4f74-94c6-361c0662716b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:53:38.157931 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.157819 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrqwl\" (UniqueName: \"kubernetes.io/projected/6c560829-549a-4f74-94c6-361c0662716b-kube-api-access-hrqwl\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:53:38.157931 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.157855 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c560829-549a-4f74-94c6-361c0662716b-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:53:38.157931 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.157886 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c560829-549a-4f74-94c6-361c0662716b-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:53:38.157931 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.157898 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c560829-549a-4f74-94c6-361c0662716b-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:53:38.595439 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.595401 4965 generic.go:356] "Generic (PLEG): container finished" podID="6c560829-549a-4f74-94c6-361c0662716b" containerID="3d46b129c731f4188dff5526f5df3deeb65137001d43d7f55494821423719b9a" exitCode=0 Oct 30 17:53:38.595663 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.595455 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" Oct 30 17:53:38.595663 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.595477 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" event={"ID":"6c560829-549a-4f74-94c6-361c0662716b","Type":"ContainerDied","Data":"3d46b129c731f4188dff5526f5df3deeb65137001d43d7f55494821423719b9a"} Oct 30 17:53:38.595663 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.595535 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2" event={"ID":"6c560829-549a-4f74-94c6-361c0662716b","Type":"ContainerDied","Data":"8012f320df6fd98c24de56b3521e5bf512b54d665953356011891af86fbd3eab"} Oct 30 17:53:38.595663 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.595557 4965 scope.go:122] "RemoveContainer" containerID="3d46b129c731f4188dff5526f5df3deeb65137001d43d7f55494821423719b9a" Oct 30 17:53:38.606939 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.606919 4965 scope.go:122] "RemoveContainer" containerID="3d46b129c731f4188dff5526f5df3deeb65137001d43d7f55494821423719b9a" Oct 30 17:53:38.607243 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:53:38.607216 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d46b129c731f4188dff5526f5df3deeb65137001d43d7f55494821423719b9a\": container with ID starting with 3d46b129c731f4188dff5526f5df3deeb65137001d43d7f55494821423719b9a not found: ID does not exist" containerID="3d46b129c731f4188dff5526f5df3deeb65137001d43d7f55494821423719b9a" Oct 30 17:53:38.607344 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.607249 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d46b129c731f4188dff5526f5df3deeb65137001d43d7f55494821423719b9a"} err="failed to get container status \"3d46b129c731f4188dff5526f5df3deeb65137001d43d7f55494821423719b9a\": rpc error: code = NotFound desc = could not find container \"3d46b129c731f4188dff5526f5df3deeb65137001d43d7f55494821423719b9a\": container with ID starting with 3d46b129c731f4188dff5526f5df3deeb65137001d43d7f55494821423719b9a not found: ID does not exist" Oct 30 17:53:38.661787 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.661756 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2"] Oct 30 17:53:38.675424 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:38.675393 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-p64q2"] Oct 30 17:53:40.101757 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:40.101722 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c560829-549a-4f74-94c6-361c0662716b" path="/var/lib/kubelet/pods/6c560829-549a-4f74-94c6-361c0662716b/volumes" Oct 30 17:53:42.752731 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:42.752697 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9"] Oct 30 17:53:42.753543 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:42.753523 4965 cpu_manager.go:414] "containerMap: removing container" podUID="6c560829-549a-4f74-94c6-361c0662716b" containerName="istio-proxy" Oct 30 17:53:42.753543 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:42.753541 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c560829-549a-4f74-94c6-361c0662716b" containerName="istio-proxy" Oct 30 17:53:42.753696 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:42.753676 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c560829-549a-4f74-94c6-361c0662716b" containerName="istio-proxy" Oct 30 17:53:42.758978 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:42.758956 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9" Oct 30 17:53:42.777580 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:42.777558 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9"] Oct 30 17:53:42.782562 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:42.782540 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-23c1e8-serving-cert\"" Oct 30 17:53:42.830564 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:42.830513 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r9ss\" (UniqueName: \"kubernetes.io/projected/40110c67-890b-40ea-ad4c-0c673666f1cd-kube-api-access-9r9ss\") pod \"maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9\" (UID: \"40110c67-890b-40ea-ad4c-0c673666f1cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9" Oct 30 17:53:42.830926 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:42.830902 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40110c67-890b-40ea-ad4c-0c673666f1cd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9\" (UID: \"40110c67-890b-40ea-ad4c-0c673666f1cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9" Oct 30 17:53:42.831591 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:42.831569 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40110c67-890b-40ea-ad4c-0c673666f1cd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9\" (UID: \"40110c67-890b-40ea-ad4c-0c673666f1cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9" Oct 30 17:53:42.831906 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:42.831833 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40110c67-890b-40ea-ad4c-0c673666f1cd-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9\" (UID: \"40110c67-890b-40ea-ad4c-0c673666f1cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9" Oct 30 17:53:42.933610 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:42.933579 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r9ss\" (UniqueName: \"kubernetes.io/projected/40110c67-890b-40ea-ad4c-0c673666f1cd-kube-api-access-9r9ss\") pod \"maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9\" (UID: \"40110c67-890b-40ea-ad4c-0c673666f1cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9" Oct 30 17:53:42.933810 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:42.933686 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40110c67-890b-40ea-ad4c-0c673666f1cd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9\" (UID: \"40110c67-890b-40ea-ad4c-0c673666f1cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9" Oct 30 17:53:42.933810 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:42.933719 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40110c67-890b-40ea-ad4c-0c673666f1cd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9\" (UID: \"40110c67-890b-40ea-ad4c-0c673666f1cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9" Oct 30 17:53:42.933965 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:42.933839 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40110c67-890b-40ea-ad4c-0c673666f1cd-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9\" (UID: \"40110c67-890b-40ea-ad4c-0c673666f1cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9" Oct 30 17:53:42.934547 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:42.934523 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40110c67-890b-40ea-ad4c-0c673666f1cd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9\" (UID: \"40110c67-890b-40ea-ad4c-0c673666f1cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9" Oct 30 17:53:42.936490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:42.936466 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40110c67-890b-40ea-ad4c-0c673666f1cd-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9\" (UID: \"40110c67-890b-40ea-ad4c-0c673666f1cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9" Oct 30 17:53:42.936580 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:42.936533 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40110c67-890b-40ea-ad4c-0c673666f1cd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9\" (UID: \"40110c67-890b-40ea-ad4c-0c673666f1cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9" Oct 30 17:53:42.957166 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:42.957140 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r9ss\" (UniqueName: \"kubernetes.io/projected/40110c67-890b-40ea-ad4c-0c673666f1cd-kube-api-access-9r9ss\") pod \"maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9\" (UID: \"40110c67-890b-40ea-ad4c-0c673666f1cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9" Oct 30 17:53:43.076948 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:43.076693 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9" Oct 30 17:53:43.432909 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:43.432881 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9"] Oct 30 17:53:43.433117 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:53:43.433090 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40110c67_890b_40ea_ad4c_0c673666f1cd.slice/crio-433227b4642c2bb5798af760062f8711cc236af8a2752f69e21ef588c7224ce0 WatchSource:0}: Error finding container 433227b4642c2bb5798af760062f8711cc236af8a2752f69e21ef588c7224ce0: Status 404 returned error can't find the container with id 433227b4642c2bb5798af760062f8711cc236af8a2752f69e21ef588c7224ce0 Oct 30 17:53:43.434832 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:43.434812 4965 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:53:43.644316 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:43.644281 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9" event={"ID":"40110c67-890b-40ea-ad4c-0c673666f1cd","Type":"ContainerStarted","Data":"433227b4642c2bb5798af760062f8711cc236af8a2752f69e21ef588c7224ce0"} Oct 30 17:53:44.654656 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:44.654621 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9" event={"ID":"40110c67-890b-40ea-ad4c-0c673666f1cd","Type":"ContainerStarted","Data":"c5f43f964ad914c9d47741e9e207aab7d5ffea95efad6766e1cf06d4c1fdb72c"} Oct 30 17:53:44.655217 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:44.654782 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9" Oct 30 17:53:50.671339 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:50.671309 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9" Oct 30 17:53:50.717872 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:53:50.717755 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9" podStartSLOduration=8.317963773 podStartE2EDuration="8.717738598s" totalImagesPullingTime="399.774825ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:53:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:53:44.717180584 +0000 UTC m=+2482.769280516" watchObservedRunningTime="2026-10-30 17:53:50.717738598 +0000 UTC m=+2488.769838531" Oct 30 17:54:02.171409 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:02.170372 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:54:06.000012 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:54:05.999975 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:54:13.517156 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:13.517123 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd"] Oct 30 17:54:13.517796 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:13.517402 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd" podUID="d98ac6ce-ffe4-430f-849e-91daa3b32273" containerName="maas-api" containerID="cri-o://2fc38dd7995d081ef6a155e5c28e1966fe84b04476c22294bfd0be5d7ecdbb57" gracePeriod=30 Oct 30 17:54:14.030028 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:14.029990 4965 generic.go:356] "Generic (PLEG): container finished" podID="d98ac6ce-ffe4-430f-849e-91daa3b32273" containerID="2fc38dd7995d081ef6a155e5c28e1966fe84b04476c22294bfd0be5d7ecdbb57" exitCode=0 Oct 30 17:54:14.030512 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:14.030186 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd" Oct 30 17:54:14.033945 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:14.030212 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd" event={"ID":"d98ac6ce-ffe4-430f-849e-91daa3b32273","Type":"ContainerDied","Data":"2fc38dd7995d081ef6a155e5c28e1966fe84b04476c22294bfd0be5d7ecdbb57"} Oct 30 17:54:14.033945 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:14.032389 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd" event={"ID":"d98ac6ce-ffe4-430f-849e-91daa3b32273","Type":"ContainerDied","Data":"ae4e2da74956329a76b4d9f91fc5ea9f9a11b6fc73c3e43e02642be4f72b6cd4"} Oct 30 17:54:14.033945 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:14.032429 4965 scope.go:122] "RemoveContainer" containerID="2fc38dd7995d081ef6a155e5c28e1966fe84b04476c22294bfd0be5d7ecdbb57" Oct 30 17:54:14.105968 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:14.105940 4965 scope.go:122] "RemoveContainer" containerID="2fc38dd7995d081ef6a155e5c28e1966fe84b04476c22294bfd0be5d7ecdbb57" Oct 30 17:54:14.112162 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:54:14.108416 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fc38dd7995d081ef6a155e5c28e1966fe84b04476c22294bfd0be5d7ecdbb57\": container with ID starting with 2fc38dd7995d081ef6a155e5c28e1966fe84b04476c22294bfd0be5d7ecdbb57 not found: ID does not exist" containerID="2fc38dd7995d081ef6a155e5c28e1966fe84b04476c22294bfd0be5d7ecdbb57" Oct 30 17:54:14.112162 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:14.108459 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc38dd7995d081ef6a155e5c28e1966fe84b04476c22294bfd0be5d7ecdbb57"} err="failed to get container status \"2fc38dd7995d081ef6a155e5c28e1966fe84b04476c22294bfd0be5d7ecdbb57\": rpc error: code = NotFound desc = could not find container \"2fc38dd7995d081ef6a155e5c28e1966fe84b04476c22294bfd0be5d7ecdbb57\": container with ID starting with 2fc38dd7995d081ef6a155e5c28e1966fe84b04476c22294bfd0be5d7ecdbb57 not found: ID does not exist" Oct 30 17:54:14.168259 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:14.168217 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d98ac6ce-ffe4-430f-849e-91daa3b32273-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d98ac6ce-ffe4-430f-849e-91daa3b32273-metrics-certs\") pod \"d98ac6ce-ffe4-430f-849e-91daa3b32273\" (UID: \"d98ac6ce-ffe4-430f-849e-91daa3b32273\") " Oct 30 17:54:14.168472 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:14.168336 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d98ac6ce-ffe4-430f-849e-91daa3b32273-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d98ac6ce-ffe4-430f-849e-91daa3b32273-openshift-service-ca\") pod \"d98ac6ce-ffe4-430f-849e-91daa3b32273\" (UID: \"d98ac6ce-ffe4-430f-849e-91daa3b32273\") " Oct 30 17:54:14.168588 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:14.168566 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d98ac6ce-ffe4-430f-849e-91daa3b32273-kube-api-access-jsvdr\" (UniqueName: \"kubernetes.io/projected/d98ac6ce-ffe4-430f-849e-91daa3b32273-kube-api-access-jsvdr\") pod \"d98ac6ce-ffe4-430f-849e-91daa3b32273\" (UID: \"d98ac6ce-ffe4-430f-849e-91daa3b32273\") " Oct 30 17:54:14.168720 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:14.168654 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d98ac6ce-ffe4-430f-849e-91daa3b32273-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d98ac6ce-ffe4-430f-849e-91daa3b32273-maas-api-tls\") pod \"d98ac6ce-ffe4-430f-849e-91daa3b32273\" (UID: \"d98ac6ce-ffe4-430f-849e-91daa3b32273\") " Oct 30 17:54:14.169907 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:14.169878 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d98ac6ce-ffe4-430f-849e-91daa3b32273-openshift-service-ca" pod "d98ac6ce-ffe4-430f-849e-91daa3b32273" (UID: "d98ac6ce-ffe4-430f-849e-91daa3b32273"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:54:14.172179 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:14.172148 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d98ac6ce-ffe4-430f-849e-91daa3b32273-metrics-certs" pod "d98ac6ce-ffe4-430f-849e-91daa3b32273" (UID: "d98ac6ce-ffe4-430f-849e-91daa3b32273"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:54:14.172481 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:14.172458 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d98ac6ce-ffe4-430f-849e-91daa3b32273-maas-api-tls" pod "d98ac6ce-ffe4-430f-849e-91daa3b32273" (UID: "d98ac6ce-ffe4-430f-849e-91daa3b32273"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:54:14.173333 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:14.173307 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98ac6ce-ffe4-430f-849e-91daa3b32273-kube-api-access-jsvdr" pod "d98ac6ce-ffe4-430f-849e-91daa3b32273" (UID: "d98ac6ce-ffe4-430f-849e-91daa3b32273"). InnerVolumeSpecName "kube-api-access-jsvdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:54:14.270842 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:14.270801 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsvdr\" (UniqueName: \"kubernetes.io/projected/d98ac6ce-ffe4-430f-849e-91daa3b32273-kube-api-access-jsvdr\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:14.270842 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:14.270835 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d98ac6ce-ffe4-430f-849e-91daa3b32273-maas-api-tls\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:14.270842 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:14.270848 4965 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d98ac6ce-ffe4-430f-849e-91daa3b32273-metrics-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:14.271110 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:14.270875 4965 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d98ac6ce-ffe4-430f-849e-91daa3b32273-openshift-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:15.039183 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:15.039149 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd" Oct 30 17:54:15.091392 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:15.091355 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd"] Oct 30 17:54:15.103622 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:15.103589 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-043253-f477d4f4d-bnwxd"] Oct 30 17:54:16.101045 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:16.101015 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d98ac6ce-ffe4-430f-849e-91daa3b32273" path="/var/lib/kubelet/pods/d98ac6ce-ffe4-430f-849e-91daa3b32273/volumes" Oct 30 17:54:21.832943 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:21.832905 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx"] Oct 30 17:54:21.833577 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:21.833160 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" podUID="f98b7bc7-2701-4a7e-9847-131fb50d65c9" containerName="istio-proxy" containerID="cri-o://dd484f82b92d6ef0691510f7730c95b4bee030409a73e9bad066c8cfa999ca48" gracePeriod=30 Oct 30 17:54:21.850984 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:21.850955 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz"] Oct 30 17:54:21.851366 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:21.851336 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" podUID="93dead59-30a5-4adb-a6f1-e54a36ff320f" containerName="istio-proxy" containerID="cri-o://50a1b36a7c53a2eb33152db0525f23b8a901458853add1f4accec2ffe9b15378" gracePeriod=30 Oct 30 17:54:21.877937 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:21.877903 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk"] Oct 30 17:54:21.878444 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:21.878409 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" podUID="21aa3bba-8be7-4766-b3ed-2c4684ccf614" containerName="istio-proxy" containerID="cri-o://62758005a1ef6a357a1a5c698d5e6fbdbceacf8e6b79add2877f6b88f8d0d7e4" gracePeriod=30 Oct 30 17:54:27.191928 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.191902 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:54:27.195073 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.194829 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:54:27.240197 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.240162 4965 generic.go:356] "Generic (PLEG): container finished" podID="f98b7bc7-2701-4a7e-9847-131fb50d65c9" containerID="dd484f82b92d6ef0691510f7730c95b4bee030409a73e9bad066c8cfa999ca48" exitCode=0 Oct 30 17:54:27.240381 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.240264 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" Oct 30 17:54:27.240381 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.240265 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" event={"ID":"f98b7bc7-2701-4a7e-9847-131fb50d65c9","Type":"ContainerDied","Data":"dd484f82b92d6ef0691510f7730c95b4bee030409a73e9bad066c8cfa999ca48"} Oct 30 17:54:27.240513 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.240419 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx" event={"ID":"f98b7bc7-2701-4a7e-9847-131fb50d65c9","Type":"ContainerDied","Data":"2cb8cc76fba2af46277ce7b1c5ddf63a951b644fd8882b784bc3bd77b70ce668"} Oct 30 17:54:27.240513 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.240447 4965 scope.go:122] "RemoveContainer" containerID="dd484f82b92d6ef0691510f7730c95b4bee030409a73e9bad066c8cfa999ca48" Oct 30 17:54:27.244189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.244159 4965 generic.go:356] "Generic (PLEG): container finished" podID="93dead59-30a5-4adb-a6f1-e54a36ff320f" containerID="50a1b36a7c53a2eb33152db0525f23b8a901458853add1f4accec2ffe9b15378" exitCode=0 Oct 30 17:54:27.244351 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.244239 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" event={"ID":"93dead59-30a5-4adb-a6f1-e54a36ff320f","Type":"ContainerDied","Data":"50a1b36a7c53a2eb33152db0525f23b8a901458853add1f4accec2ffe9b15378"} Oct 30 17:54:27.246755 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.246273 4965 generic.go:356] "Generic (PLEG): container finished" podID="21aa3bba-8be7-4766-b3ed-2c4684ccf614" containerID="62758005a1ef6a357a1a5c698d5e6fbdbceacf8e6b79add2877f6b88f8d0d7e4" exitCode=0 Oct 30 17:54:27.246755 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.246461 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" event={"ID":"21aa3bba-8be7-4766-b3ed-2c4684ccf614","Type":"ContainerDied","Data":"62758005a1ef6a357a1a5c698d5e6fbdbceacf8e6b79add2877f6b88f8d0d7e4"} Oct 30 17:54:27.246755 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.246502 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" event={"ID":"21aa3bba-8be7-4766-b3ed-2c4684ccf614","Type":"ContainerDied","Data":"e659db17d9460a5d6fac612338877dc788afa65e6eda135b0cb7f680ccc69c98"} Oct 30 17:54:27.246755 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.246582 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk" Oct 30 17:54:27.259167 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.257252 4965 scope.go:122] "RemoveContainer" containerID="dd484f82b92d6ef0691510f7730c95b4bee030409a73e9bad066c8cfa999ca48" Oct 30 17:54:27.259167 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:54:27.257662 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd484f82b92d6ef0691510f7730c95b4bee030409a73e9bad066c8cfa999ca48\": container with ID starting with dd484f82b92d6ef0691510f7730c95b4bee030409a73e9bad066c8cfa999ca48 not found: ID does not exist" containerID="dd484f82b92d6ef0691510f7730c95b4bee030409a73e9bad066c8cfa999ca48" Oct 30 17:54:27.259167 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.257699 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd484f82b92d6ef0691510f7730c95b4bee030409a73e9bad066c8cfa999ca48"} err="failed to get container status \"dd484f82b92d6ef0691510f7730c95b4bee030409a73e9bad066c8cfa999ca48\": rpc error: code = NotFound desc = could not find container \"dd484f82b92d6ef0691510f7730c95b4bee030409a73e9bad066c8cfa999ca48\": container with ID starting with dd484f82b92d6ef0691510f7730c95b4bee030409a73e9bad066c8cfa999ca48 not found: ID does not exist" Oct 30 17:54:27.259167 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.257723 4965 scope.go:122] "RemoveContainer" containerID="62758005a1ef6a357a1a5c698d5e6fbdbceacf8e6b79add2877f6b88f8d0d7e4" Oct 30 17:54:27.283750 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.283556 4965 scope.go:122] "RemoveContainer" containerID="62758005a1ef6a357a1a5c698d5e6fbdbceacf8e6b79add2877f6b88f8d0d7e4" Oct 30 17:54:27.284470 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:54:27.284369 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62758005a1ef6a357a1a5c698d5e6fbdbceacf8e6b79add2877f6b88f8d0d7e4\": container with ID starting with 62758005a1ef6a357a1a5c698d5e6fbdbceacf8e6b79add2877f6b88f8d0d7e4 not found: ID does not exist" containerID="62758005a1ef6a357a1a5c698d5e6fbdbceacf8e6b79add2877f6b88f8d0d7e4" Oct 30 17:54:27.284470 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.284407 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62758005a1ef6a357a1a5c698d5e6fbdbceacf8e6b79add2877f6b88f8d0d7e4"} err="failed to get container status \"62758005a1ef6a357a1a5c698d5e6fbdbceacf8e6b79add2877f6b88f8d0d7e4\": rpc error: code = NotFound desc = could not find container \"62758005a1ef6a357a1a5c698d5e6fbdbceacf8e6b79add2877f6b88f8d0d7e4\": container with ID starting with 62758005a1ef6a357a1a5c698d5e6fbdbceacf8e6b79add2877f6b88f8d0d7e4 not found: ID does not exist" Oct 30 17:54:27.286600 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.286575 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:54:27.307042 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.307007 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93dead59-30a5-4adb-a6f1-e54a36ff320f-istio-token\" (UniqueName: \"kubernetes.io/projected/93dead59-30a5-4adb-a6f1-e54a36ff320f-istio-token\") pod \"93dead59-30a5-4adb-a6f1-e54a36ff320f\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " Oct 30 17:54:27.307194 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.307069 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istiod-ca-cert\") pod \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " Oct 30 17:54:27.307194 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.307105 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93dead59-30a5-4adb-a6f1-e54a36ff320f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93dead59-30a5-4adb-a6f1-e54a36ff320f-istio-podinfo\") pod \"93dead59-30a5-4adb-a6f1-e54a36ff320f\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " Oct 30 17:54:27.307194 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.307142 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-workload-socket\") pod \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " Oct 30 17:54:27.307194 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.307173 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istio-token\" (UniqueName: \"kubernetes.io/projected/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istio-token\") pod \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " Oct 30 17:54:27.307430 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.307221 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istio-data\") pod \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " Oct 30 17:54:27.307430 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.307254 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-workload-socket\") pod \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " Oct 30 17:54:27.307430 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.307282 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21aa3bba-8be7-4766-b3ed-2c4684ccf614-kube-api-access-k7f42\" (UniqueName: \"kubernetes.io/projected/21aa3bba-8be7-4766-b3ed-2c4684ccf614-kube-api-access-k7f42\") pod \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " Oct 30 17:54:27.307430 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.307322 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istio-envoy\") pod \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " Oct 30 17:54:27.307430 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.307348 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-workload-socket\") pod \"93dead59-30a5-4adb-a6f1-e54a36ff320f\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " Oct 30 17:54:27.307430 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.307386 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93dead59-30a5-4adb-a6f1-e54a36ff320f-kube-api-access-wrpnm\" (UniqueName: \"kubernetes.io/projected/93dead59-30a5-4adb-a6f1-e54a36ff320f-kube-api-access-wrpnm\") pod \"93dead59-30a5-4adb-a6f1-e54a36ff320f\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " Oct 30 17:54:27.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.307458 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-credential-socket\") pod \"93dead59-30a5-4adb-a6f1-e54a36ff320f\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " Oct 30 17:54:27.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.307530 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-workload-certs\") pod \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " Oct 30 17:54:27.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.307578 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istio-envoy\") pod \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " Oct 30 17:54:27.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.307603 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-istio-data\") pod \"93dead59-30a5-4adb-a6f1-e54a36ff320f\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " Oct 30 17:54:27.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.307642 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istio-podinfo\") pod \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " Oct 30 17:54:27.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.307680 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istiod-ca-cert" pod "f98b7bc7-2701-4a7e-9847-131fb50d65c9" (UID: "f98b7bc7-2701-4a7e-9847-131fb50d65c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:54:27.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.307698 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-credential-socket\") pod \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " Oct 30 17:54:27.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.307731 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istio-token\" (UniqueName: \"kubernetes.io/projected/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istio-token\") pod \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " Oct 30 17:54:27.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.307761 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-workload-certs\") pod \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " Oct 30 17:54:27.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.307818 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-istio-envoy\") pod \"93dead59-30a5-4adb-a6f1-e54a36ff320f\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " Oct 30 17:54:27.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.307929 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-workload-certs\") pod \"93dead59-30a5-4adb-a6f1-e54a36ff320f\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " Oct 30 17:54:27.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.307985 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-credential-socket\") pod \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " Oct 30 17:54:27.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.308022 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-credential-socket" pod "93dead59-30a5-4adb-a6f1-e54a36ff320f" (UID: "93dead59-30a5-4adb-a6f1-e54a36ff320f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:27.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.308070 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istio-podinfo\") pod \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " Oct 30 17:54:27.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.308101 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93dead59-30a5-4adb-a6f1-e54a36ff320f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93dead59-30a5-4adb-a6f1-e54a36ff320f-istiod-ca-cert\") pod \"93dead59-30a5-4adb-a6f1-e54a36ff320f\" (UID: \"93dead59-30a5-4adb-a6f1-e54a36ff320f\") " Oct 30 17:54:27.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.308152 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f98b7bc7-2701-4a7e-9847-131fb50d65c9-kube-api-access-9xn6n\" (UniqueName: \"kubernetes.io/projected/f98b7bc7-2701-4a7e-9847-131fb50d65c9-kube-api-access-9xn6n\") pod \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " Oct 30 17:54:27.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.308254 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istio-data\") pod \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\" (UID: \"f98b7bc7-2701-4a7e-9847-131fb50d65c9\") " Oct 30 17:54:27.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.308281 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istiod-ca-cert\") pod \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\" (UID: \"21aa3bba-8be7-4766-b3ed-2c4684ccf614\") " Oct 30 17:54:27.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.308533 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-workload-certs" pod "f98b7bc7-2701-4a7e-9847-131fb50d65c9" (UID: "f98b7bc7-2701-4a7e-9847-131fb50d65c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:27.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.308702 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-workload-certs" pod "21aa3bba-8be7-4766-b3ed-2c4684ccf614" (UID: "21aa3bba-8be7-4766-b3ed-2c4684ccf614"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:27.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.309198 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.309222 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.309241 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.309255 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.309770 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-workload-socket" pod "93dead59-30a5-4adb-a6f1-e54a36ff320f" (UID: "93dead59-30a5-4adb-a6f1-e54a36ff320f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:27.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.310149 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istio-data" pod "21aa3bba-8be7-4766-b3ed-2c4684ccf614" (UID: "21aa3bba-8be7-4766-b3ed-2c4684ccf614"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:27.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.310217 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-workload-socket" pod "21aa3bba-8be7-4766-b3ed-2c4684ccf614" (UID: "21aa3bba-8be7-4766-b3ed-2c4684ccf614"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:27.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.310443 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istio-data" pod "f98b7bc7-2701-4a7e-9847-131fb50d65c9" (UID: "f98b7bc7-2701-4a7e-9847-131fb50d65c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:27.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.310590 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93dead59-30a5-4adb-a6f1-e54a36ff320f-istiod-ca-cert" pod "93dead59-30a5-4adb-a6f1-e54a36ff320f" (UID: "93dead59-30a5-4adb-a6f1-e54a36ff320f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:54:27.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.310931 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-workload-certs" pod "93dead59-30a5-4adb-a6f1-e54a36ff320f" (UID: "93dead59-30a5-4adb-a6f1-e54a36ff320f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:27.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.311196 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-credential-socket" pod "f98b7bc7-2701-4a7e-9847-131fb50d65c9" (UID: "f98b7bc7-2701-4a7e-9847-131fb50d65c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:27.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.311204 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istiod-ca-cert" pod "21aa3bba-8be7-4766-b3ed-2c4684ccf614" (UID: "21aa3bba-8be7-4766-b3ed-2c4684ccf614"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:54:27.313004 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.311472 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-workload-socket" pod "f98b7bc7-2701-4a7e-9847-131fb50d65c9" (UID: "f98b7bc7-2701-4a7e-9847-131fb50d65c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:27.313004 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.311820 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-istio-data" pod "93dead59-30a5-4adb-a6f1-e54a36ff320f" (UID: "93dead59-30a5-4adb-a6f1-e54a36ff320f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:27.316157 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.316128 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-credential-socket" pod "21aa3bba-8be7-4766-b3ed-2c4684ccf614" (UID: "21aa3bba-8be7-4766-b3ed-2c4684ccf614"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:27.317489 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.317439 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istio-podinfo" pod "21aa3bba-8be7-4766-b3ed-2c4684ccf614" (UID: "21aa3bba-8be7-4766-b3ed-2c4684ccf614"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:54:27.317489 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.317456 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93dead59-30a5-4adb-a6f1-e54a36ff320f-istio-podinfo" pod "93dead59-30a5-4adb-a6f1-e54a36ff320f" (UID: "93dead59-30a5-4adb-a6f1-e54a36ff320f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:54:27.317641 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.317504 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-istio-envoy" pod "93dead59-30a5-4adb-a6f1-e54a36ff320f" (UID: "93dead59-30a5-4adb-a6f1-e54a36ff320f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:27.318333 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.318246 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istio-token" pod "f98b7bc7-2701-4a7e-9847-131fb50d65c9" (UID: "f98b7bc7-2701-4a7e-9847-131fb50d65c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:54:27.318333 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.318246 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istio-envoy" pod "21aa3bba-8be7-4766-b3ed-2c4684ccf614" (UID: "21aa3bba-8be7-4766-b3ed-2c4684ccf614"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:27.318333 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.318295 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istio-token" pod "21aa3bba-8be7-4766-b3ed-2c4684ccf614" (UID: "21aa3bba-8be7-4766-b3ed-2c4684ccf614"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:54:27.318333 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.318295 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21aa3bba-8be7-4766-b3ed-2c4684ccf614-kube-api-access-k7f42" pod "21aa3bba-8be7-4766-b3ed-2c4684ccf614" (UID: "21aa3bba-8be7-4766-b3ed-2c4684ccf614"). InnerVolumeSpecName "kube-api-access-k7f42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:54:27.318333 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.318312 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istio-envoy" pod "f98b7bc7-2701-4a7e-9847-131fb50d65c9" (UID: "f98b7bc7-2701-4a7e-9847-131fb50d65c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:27.318946 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.318921 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93dead59-30a5-4adb-a6f1-e54a36ff320f-istio-token" pod "93dead59-30a5-4adb-a6f1-e54a36ff320f" (UID: "93dead59-30a5-4adb-a6f1-e54a36ff320f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:54:27.319037 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.318968 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f98b7bc7-2701-4a7e-9847-131fb50d65c9-kube-api-access-9xn6n" pod "f98b7bc7-2701-4a7e-9847-131fb50d65c9" (UID: "f98b7bc7-2701-4a7e-9847-131fb50d65c9"). InnerVolumeSpecName "kube-api-access-9xn6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:54:27.319157 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.319134 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istio-podinfo" pod "f98b7bc7-2701-4a7e-9847-131fb50d65c9" (UID: "f98b7bc7-2701-4a7e-9847-131fb50d65c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:54:27.320413 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.320387 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93dead59-30a5-4adb-a6f1-e54a36ff320f-kube-api-access-wrpnm" pod "93dead59-30a5-4adb-a6f1-e54a36ff320f" (UID: "93dead59-30a5-4adb-a6f1-e54a36ff320f"). InnerVolumeSpecName "kube-api-access-wrpnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:54:27.410777 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.410692 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.410777 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.410718 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.410777 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.410727 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.410777 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.410735 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.410777 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.410745 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.410777 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.410754 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.410777 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.410761 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.410777 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.410770 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.410777 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.410779 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.411206 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.410787 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93dead59-30a5-4adb-a6f1-e54a36ff320f-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.411206 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.410797 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xn6n\" (UniqueName: \"kubernetes.io/projected/f98b7bc7-2701-4a7e-9847-131fb50d65c9-kube-api-access-9xn6n\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.411206 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.410804 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.411206 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.410813 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.411206 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.410820 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93dead59-30a5-4adb-a6f1-e54a36ff320f-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.411206 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.410828 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93dead59-30a5-4adb-a6f1-e54a36ff320f-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.411206 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.410835 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98b7bc7-2701-4a7e-9847-131fb50d65c9-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.411206 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.410844 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.411206 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.410851 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.411206 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.410874 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.411206 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.410887 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7f42\" (UniqueName: \"kubernetes.io/projected/21aa3bba-8be7-4766-b3ed-2c4684ccf614-kube-api-access-k7f42\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.411206 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.410899 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21aa3bba-8be7-4766-b3ed-2c4684ccf614-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.411206 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.410912 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93dead59-30a5-4adb-a6f1-e54a36ff320f-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.411206 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.410921 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrpnm\" (UniqueName: \"kubernetes.io/projected/93dead59-30a5-4adb-a6f1-e54a36ff320f-kube-api-access-wrpnm\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.589457 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.589423 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx"] Oct 30 17:54:27.595072 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.595046 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-bjjmx"] Oct 30 17:54:27.630422 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.630389 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk"] Oct 30 17:54:27.633819 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:27.633800 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-nrfqk"] Oct 30 17:54:28.112019 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:28.111987 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21aa3bba-8be7-4766-b3ed-2c4684ccf614" path="/var/lib/kubelet/pods/21aa3bba-8be7-4766-b3ed-2c4684ccf614/volumes" Oct 30 17:54:28.112528 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:28.112514 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f98b7bc7-2701-4a7e-9847-131fb50d65c9" path="/var/lib/kubelet/pods/f98b7bc7-2701-4a7e-9847-131fb50d65c9/volumes" Oct 30 17:54:28.257313 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:28.257284 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" Oct 30 17:54:28.257775 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:28.257286 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz" event={"ID":"93dead59-30a5-4adb-a6f1-e54a36ff320f","Type":"ContainerDied","Data":"9b567489ea643fc298c2bd7aa76be6d1cf8261b5c95d37344b16bd9dedc0e8b4"} Oct 30 17:54:28.257775 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:28.257432 4965 scope.go:122] "RemoveContainer" containerID="50a1b36a7c53a2eb33152db0525f23b8a901458853add1f4accec2ffe9b15378" Oct 30 17:54:28.311189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:28.311157 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz"] Oct 30 17:54:28.328935 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:28.321209 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-fw7dz"] Oct 30 17:54:30.105232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:30.105194 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93dead59-30a5-4adb-a6f1-e54a36ff320f" path="/var/lib/kubelet/pods/93dead59-30a5-4adb-a6f1-e54a36ff320f/volumes" Oct 30 17:54:30.846483 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:30.846449 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6"] Oct 30 17:54:30.847379 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:30.847355 4965 cpu_manager.go:414] "containerMap: removing container" podUID="21aa3bba-8be7-4766-b3ed-2c4684ccf614" containerName="istio-proxy" Oct 30 17:54:30.847379 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:30.847377 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21aa3bba-8be7-4766-b3ed-2c4684ccf614" containerName="istio-proxy" Oct 30 17:54:30.847536 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:30.847413 4965 cpu_manager.go:414] "containerMap: removing container" podUID="f98b7bc7-2701-4a7e-9847-131fb50d65c9" containerName="istio-proxy" Oct 30 17:54:30.847536 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:30.847420 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f98b7bc7-2701-4a7e-9847-131fb50d65c9" containerName="istio-proxy" Oct 30 17:54:30.847536 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:30.847429 4965 cpu_manager.go:414] "containerMap: removing container" podUID="d98ac6ce-ffe4-430f-849e-91daa3b32273" containerName="maas-api" Oct 30 17:54:30.847536 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:30.847435 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d98ac6ce-ffe4-430f-849e-91daa3b32273" containerName="maas-api" Oct 30 17:54:30.847536 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:30.847454 4965 cpu_manager.go:414] "containerMap: removing container" podUID="93dead59-30a5-4adb-a6f1-e54a36ff320f" containerName="istio-proxy" Oct 30 17:54:30.847536 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:30.847459 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93dead59-30a5-4adb-a6f1-e54a36ff320f" containerName="istio-proxy" Oct 30 17:54:30.847724 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:30.847561 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="93dead59-30a5-4adb-a6f1-e54a36ff320f" containerName="istio-proxy" Oct 30 17:54:30.847724 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:30.847575 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="21aa3bba-8be7-4766-b3ed-2c4684ccf614" containerName="istio-proxy" Oct 30 17:54:30.847724 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:30.847586 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="f98b7bc7-2701-4a7e-9847-131fb50d65c9" containerName="istio-proxy" Oct 30 17:54:30.847724 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:30.847593 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="d98ac6ce-ffe4-430f-849e-91daa3b32273" containerName="maas-api" Oct 30 17:54:30.852936 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:30.852917 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:30.869372 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:30.869320 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-01b925-openshift-default-dockercfg-mn67p\"" Oct 30 17:54:30.873002 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:30.872977 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6"] Oct 30 17:54:30.913267 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:30.913233 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-credential-socket\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:30.913464 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:30.913369 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-workload-socket\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:30.913464 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:30.913401 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-istio-envoy\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:30.913603 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:30.913571 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46a78684-48db-4c15-9122-0591ec698f82-istio-podinfo\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:30.913753 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:30.913634 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46a78684-48db-4c15-9122-0591ec698f82-istiod-ca-cert\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:30.913753 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:30.913666 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46a78684-48db-4c15-9122-0591ec698f82-istio-token\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:30.913905 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:30.913760 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-istio-data\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:30.913905 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:30.913787 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87z6x\" (UniqueName: \"kubernetes.io/projected/46a78684-48db-4c15-9122-0591ec698f82-kube-api-access-87z6x\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:30.913905 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:30.913805 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-workload-certs\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:31.015533 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:31.015499 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46a78684-48db-4c15-9122-0591ec698f82-istio-podinfo\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:31.015737 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:31.015545 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46a78684-48db-4c15-9122-0591ec698f82-istiod-ca-cert\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:31.015737 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:31.015574 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46a78684-48db-4c15-9122-0591ec698f82-istio-token\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:31.015737 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:31.015647 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-istio-data\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:31.015737 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:31.015675 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87z6x\" (UniqueName: \"kubernetes.io/projected/46a78684-48db-4c15-9122-0591ec698f82-kube-api-access-87z6x\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:31.015737 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:31.015696 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-workload-certs\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:31.016107 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:31.016043 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-credential-socket\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:31.016226 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:31.016107 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-istio-data\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:31.016226 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:31.016174 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-workload-certs\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:31.016320 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:31.016217 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-workload-socket\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:31.016320 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:31.016263 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-istio-envoy\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:31.016416 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:31.016379 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-credential-socket\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:31.016456 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:31.016414 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46a78684-48db-4c15-9122-0591ec698f82-istiod-ca-cert\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:31.016764 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:31.016743 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-workload-socket\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:31.018261 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:31.018235 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46a78684-48db-4c15-9122-0591ec698f82-istio-podinfo\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:31.018476 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:31.018456 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-istio-envoy\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:31.040135 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:31.040108 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46a78684-48db-4c15-9122-0591ec698f82-istio-token\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:31.040245 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:31.040230 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87z6x\" (UniqueName: \"kubernetes.io/projected/46a78684-48db-4c15-9122-0591ec698f82-kube-api-access-87z6x\") pod \"e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:31.168130 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:31.168049 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:31.498512 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:31.498476 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6"] Oct 30 17:54:31.498739 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:54:31.498714 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46a78684_48db_4c15_9122_0591ec698f82.slice/crio-c5d117d27498b5ac84ebe56a0550cdabbbfbb1e62a6141abf572f549129a3ffa WatchSource:0}: Error finding container c5d117d27498b5ac84ebe56a0550cdabbbfbb1e62a6141abf572f549129a3ffa: Status 404 returned error can't find the container with id c5d117d27498b5ac84ebe56a0550cdabbbfbb1e62a6141abf572f549129a3ffa Oct 30 17:54:31.502037 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:31.502000 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:54:31.502130 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:31.502084 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:54:31.502178 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:31.502123 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:54:32.344917 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:32.344843 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" event={"ID":"46a78684-48db-4c15-9122-0591ec698f82","Type":"ContainerStarted","Data":"cedd568980dd911b8cae6aace8ecfbc513a1f7fc8aa27afd44edb23efe5185bf"} Oct 30 17:54:32.345312 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:32.344944 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" event={"ID":"46a78684-48db-4c15-9122-0591ec698f82","Type":"ContainerStarted","Data":"c5d117d27498b5ac84ebe56a0550cdabbbfbb1e62a6141abf572f549129a3ffa"} Oct 30 17:54:32.879887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:32.879772 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" podStartSLOduration=2.879748823 podStartE2EDuration="2.879748823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:54:32.428010653 +0000 UTC m=+2530.480110584" watchObservedRunningTime="2026-10-30 17:54:32.879748823 +0000 UTC m=+2530.931848757" Oct 30 17:54:32.883355 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:32.883317 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bcdac0-698d7bc59-zr7ws"] Oct 30 17:54:33.209944 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:33.209322 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:33.209944 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:33.209809 4965 patch_prober.go:29] interesting pod/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 17:54:33.209944 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:33.209855 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" podUID="46a78684-48db-4c15-9122-0591ec698f82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 17:54:34.174313 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:34.174275 4965 patch_prober.go:29] interesting pod/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 17:54:34.174934 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:34.174899 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" podUID="46a78684-48db-4c15-9122-0591ec698f82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 17:54:35.021987 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:35.021947 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-d207f8-d8677dc5d-5pgjt"] Oct 30 17:54:35.173137 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:35.170970 4965 patch_prober.go:29] interesting pod/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 17:54:35.173137 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:35.171026 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" podUID="46a78684-48db-4c15-9122-0591ec698f82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 17:54:35.216415 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:35.216376 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-d207f8-d8677dc5d-5pgjt" Oct 30 17:54:35.217744 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:35.216263 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-d207f8-d8677dc5d-5pgjt"] Oct 30 17:54:35.232843 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:35.232824 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-d207f8\"" Oct 30 17:54:35.289065 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:35.288979 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-d207f8-dockercfg-p55p9\"" Oct 30 17:54:35.328893 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:35.328830 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6706cbd9-edfa-4b4d-ae4a-44a037e61efc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-d207f8-d8677dc5d-5pgjt\" (UID: \"6706cbd9-edfa-4b4d-ae4a-44a037e61efc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-d207f8-d8677dc5d-5pgjt" Oct 30 17:54:35.329071 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:35.328987 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nszdr\" (UniqueName: \"kubernetes.io/projected/6706cbd9-edfa-4b4d-ae4a-44a037e61efc-kube-api-access-nszdr\") pod \"payload-processing-e2e-shared-b-w5-d207f8-d8677dc5d-5pgjt\" (UID: \"6706cbd9-edfa-4b4d-ae4a-44a037e61efc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-d207f8-d8677dc5d-5pgjt" Oct 30 17:54:35.430096 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:35.430061 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nszdr\" (UniqueName: \"kubernetes.io/projected/6706cbd9-edfa-4b4d-ae4a-44a037e61efc-kube-api-access-nszdr\") pod \"payload-processing-e2e-shared-b-w5-d207f8-d8677dc5d-5pgjt\" (UID: \"6706cbd9-edfa-4b4d-ae4a-44a037e61efc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-d207f8-d8677dc5d-5pgjt" Oct 30 17:54:35.430297 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:35.430280 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6706cbd9-edfa-4b4d-ae4a-44a037e61efc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-d207f8-d8677dc5d-5pgjt\" (UID: \"6706cbd9-edfa-4b4d-ae4a-44a037e61efc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-d207f8-d8677dc5d-5pgjt" Oct 30 17:54:35.431119 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:35.431095 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6706cbd9-edfa-4b4d-ae4a-44a037e61efc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-d207f8-d8677dc5d-5pgjt\" (UID: \"6706cbd9-edfa-4b4d-ae4a-44a037e61efc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-d207f8-d8677dc5d-5pgjt" Oct 30 17:54:35.454168 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:35.454139 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nszdr\" (UniqueName: \"kubernetes.io/projected/6706cbd9-edfa-4b4d-ae4a-44a037e61efc-kube-api-access-nszdr\") pod \"payload-processing-e2e-shared-b-w5-d207f8-d8677dc5d-5pgjt\" (UID: \"6706cbd9-edfa-4b4d-ae4a-44a037e61efc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-d207f8-d8677dc5d-5pgjt" Oct 30 17:54:35.541384 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:35.541299 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-d207f8-d8677dc5d-5pgjt" Oct 30 17:54:35.887267 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:35.887227 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-d207f8-d8677dc5d-5pgjt"] Oct 30 17:54:35.889039 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:54:35.889011 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6706cbd9_edfa_4b4d_ae4a_44a037e61efc.slice/crio-5d20c92066c27f09f219fb15ac671043a27f4d4e6af16c70e29bf26815210252 WatchSource:0}: Error finding container 5d20c92066c27f09f219fb15ac671043a27f4d4e6af16c70e29bf26815210252: Status 404 returned error can't find the container with id 5d20c92066c27f09f219fb15ac671043a27f4d4e6af16c70e29bf26815210252 Oct 30 17:54:36.091485 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:54:36.091441 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:54:36.181941 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:36.180976 4965 patch_prober.go:29] interesting pod/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 17:54:36.181941 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:36.181053 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" podUID="46a78684-48db-4c15-9122-0591ec698f82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 17:54:36.405105 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:36.405063 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-d207f8-d8677dc5d-5pgjt" event={"ID":"6706cbd9-edfa-4b4d-ae4a-44a037e61efc","Type":"ContainerStarted","Data":"213ec240dba617b0f3d51ad3b068504c54a851a62bf770f48a19e5c665cf5794"} Oct 30 17:54:36.405580 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:36.405138 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-d207f8-d8677dc5d-5pgjt" event={"ID":"6706cbd9-edfa-4b4d-ae4a-44a037e61efc","Type":"ContainerStarted","Data":"5d20c92066c27f09f219fb15ac671043a27f4d4e6af16c70e29bf26815210252"} Oct 30 17:54:36.405580 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:36.405171 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-d207f8-d8677dc5d-5pgjt" Oct 30 17:54:36.446598 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:36.446494 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-d207f8-d8677dc5d-5pgjt" podStartSLOduration=2.44647585 podStartE2EDuration="2.44647585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:54:36.445270247 +0000 UTC m=+2534.497370182" watchObservedRunningTime="2026-10-30 17:54:36.44647585 +0000 UTC m=+2534.498575781" Oct 30 17:54:37.182284 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:37.182240 4965 patch_prober.go:29] interesting pod/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 17:54:37.182474 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:37.182319 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" podUID="46a78684-48db-4c15-9122-0591ec698f82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 17:54:37.884175 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:37.884114 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcdac0-698d7bc59-zr7ws" podUID="3af50d6f-9968-40fd-9159-b6f7078023db" containerName="payload-pre-processing" containerID="cri-o://0c437352dfdf9d9d4f918afc7883bf279eb4964a09560cbd8ea29f956bca97bb" gracePeriod=25 Oct 30 17:54:38.168398 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:38.168298 4965 patch_prober.go:29] interesting pod/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 17:54:38.168398 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:38.168355 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" podUID="46a78684-48db-4c15-9122-0591ec698f82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 17:54:39.174098 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:39.174068 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:39.174534 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:39.174516 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:39.175231 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:39.175206 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 17:54:44.538269 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:44.538226 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcdac0-698d7bc59-zr7ws" podUID="3af50d6f-9968-40fd-9159-b6f7078023db" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:54:47.416884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:47.416843 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-d207f8-d8677dc5d-5pgjt" Oct 30 17:54:49.699654 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:49.699610 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:54:54.538129 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:54.538085 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcdac0-698d7bc59-zr7ws" podUID="3af50d6f-9968-40fd-9159-b6f7078023db" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:54:55.359301 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:55.359267 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx"] Oct 30 17:54:58.188334 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:58.188304 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcdac0-698d7bc59-zr7ws" Oct 30 17:54:58.299631 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:58.299596 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3af50d6f-9968-40fd-9159-b6f7078023db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af50d6f-9968-40fd-9159-b6f7078023db-plugins-config-volume\") pod \"3af50d6f-9968-40fd-9159-b6f7078023db\" (UID: \"3af50d6f-9968-40fd-9159-b6f7078023db\") " Oct 30 17:54:58.299824 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:58.299703 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3af50d6f-9968-40fd-9159-b6f7078023db-kube-api-access-fj476\" (UniqueName: \"kubernetes.io/projected/3af50d6f-9968-40fd-9159-b6f7078023db-kube-api-access-fj476\") pod \"3af50d6f-9968-40fd-9159-b6f7078023db\" (UID: \"3af50d6f-9968-40fd-9159-b6f7078023db\") " Oct 30 17:54:58.300008 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:58.299983 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3af50d6f-9968-40fd-9159-b6f7078023db-plugins-config-volume" pod "3af50d6f-9968-40fd-9159-b6f7078023db" (UID: "3af50d6f-9968-40fd-9159-b6f7078023db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:54:58.300382 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:58.300357 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af50d6f-9968-40fd-9159-b6f7078023db-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:58.301960 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:58.301930 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af50d6f-9968-40fd-9159-b6f7078023db-kube-api-access-fj476" pod "3af50d6f-9968-40fd-9159-b6f7078023db" (UID: "3af50d6f-9968-40fd-9159-b6f7078023db"). InnerVolumeSpecName "kube-api-access-fj476". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:54:58.402203 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:58.402126 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj476\" (UniqueName: \"kubernetes.io/projected/3af50d6f-9968-40fd-9159-b6f7078023db-kube-api-access-fj476\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:54:58.613518 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:58.613481 4965 generic.go:356] "Generic (PLEG): container finished" podID="3af50d6f-9968-40fd-9159-b6f7078023db" containerID="0c437352dfdf9d9d4f918afc7883bf279eb4964a09560cbd8ea29f956bca97bb" exitCode=0 Oct 30 17:54:58.613695 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:58.613531 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcdac0-698d7bc59-zr7ws" Oct 30 17:54:58.613695 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:58.613565 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcdac0-698d7bc59-zr7ws" event={"ID":"3af50d6f-9968-40fd-9159-b6f7078023db","Type":"ContainerDied","Data":"0c437352dfdf9d9d4f918afc7883bf279eb4964a09560cbd8ea29f956bca97bb"} Oct 30 17:54:58.613695 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:58.613622 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bcdac0-698d7bc59-zr7ws" event={"ID":"3af50d6f-9968-40fd-9159-b6f7078023db","Type":"ContainerDied","Data":"e8af45149dc68249dce14c93e0c1b31af5e2ccfb25218acca0fd5b9a1361ab8b"} Oct 30 17:54:58.613695 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:58.613647 4965 scope.go:122] "RemoveContainer" containerID="0c437352dfdf9d9d4f918afc7883bf279eb4964a09560cbd8ea29f956bca97bb" Oct 30 17:54:58.624875 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:58.624844 4965 scope.go:122] "RemoveContainer" containerID="0c437352dfdf9d9d4f918afc7883bf279eb4964a09560cbd8ea29f956bca97bb" Oct 30 17:54:58.625167 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:54:58.625147 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c437352dfdf9d9d4f918afc7883bf279eb4964a09560cbd8ea29f956bca97bb\": container with ID starting with 0c437352dfdf9d9d4f918afc7883bf279eb4964a09560cbd8ea29f956bca97bb not found: ID does not exist" containerID="0c437352dfdf9d9d4f918afc7883bf279eb4964a09560cbd8ea29f956bca97bb" Oct 30 17:54:58.625218 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:58.625177 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c437352dfdf9d9d4f918afc7883bf279eb4964a09560cbd8ea29f956bca97bb"} err="failed to get container status \"0c437352dfdf9d9d4f918afc7883bf279eb4964a09560cbd8ea29f956bca97bb\": rpc error: code = NotFound desc = could not find container \"0c437352dfdf9d9d4f918afc7883bf279eb4964a09560cbd8ea29f956bca97bb\": container with ID starting with 0c437352dfdf9d9d4f918afc7883bf279eb4964a09560cbd8ea29f956bca97bb not found: ID does not exist" Oct 30 17:54:58.668501 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:58.668428 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bcdac0-698d7bc59-zr7ws"] Oct 30 17:54:58.677985 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:54:58.677955 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bcdac0-698d7bc59-zr7ws"] Oct 30 17:55:00.100254 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:00.100219 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3af50d6f-9968-40fd-9159-b6f7078023db" path="/var/lib/kubelet/pods/3af50d6f-9968-40fd-9159-b6f7078023db/volumes" Oct 30 17:55:05.997438 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:55:05.997349 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:55:08.601346 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:08.601308 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz"] Oct 30 17:55:08.602145 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:08.602111 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" podUID="50d223e1-883f-454f-9c29-348129b7d676" containerName="istio-proxy" containerID="cri-o://2ca97e7cfca2c2246fab3e6871885e92a10dffa92c8fae7a0e6576dec6495b0b" gracePeriod=30 Oct 30 17:55:10.384165 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:10.384126 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" podUID="7bd734c0-9b03-463b-adfd-e1a206fd1f9a" containerName="main" containerID="cri-o://c42c009df56ffbc9b72a033d306165886a91f90bd6ebf97edba95768b120f4c6" gracePeriod=45 Oct 30 17:55:10.492468 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:10.492432 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" podUID="7bd734c0-9b03-463b-adfd-e1a206fd1f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.173:8000/ready\": dial tcp 10.132.0.173:8000: connect: connection refused" Oct 30 17:55:10.662426 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:10.662401 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" Oct 30 17:55:10.774259 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:10.774216 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-kube-api-access-mccz9\" (UniqueName: \"kubernetes.io/projected/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-kube-api-access-mccz9\") pod \"7bd734c0-9b03-463b-adfd-e1a206fd1f9a\" (UID: \"7bd734c0-9b03-463b-adfd-e1a206fd1f9a\") " Oct 30 17:55:10.774430 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:10.774413 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-tls-certs\") pod \"7bd734c0-9b03-463b-adfd-e1a206fd1f9a\" (UID: \"7bd734c0-9b03-463b-adfd-e1a206fd1f9a\") " Oct 30 17:55:10.774473 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:10.774451 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-tmp-dir\") pod \"7bd734c0-9b03-463b-adfd-e1a206fd1f9a\" (UID: \"7bd734c0-9b03-463b-adfd-e1a206fd1f9a\") " Oct 30 17:55:10.774570 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:10.774551 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-dshm\") pod \"7bd734c0-9b03-463b-adfd-e1a206fd1f9a\" (UID: \"7bd734c0-9b03-463b-adfd-e1a206fd1f9a\") " Oct 30 17:55:10.774734 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:10.774597 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-home\" (UniqueName: \"kubernetes.io/empty-dir/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-home\") pod \"7bd734c0-9b03-463b-adfd-e1a206fd1f9a\" (UID: \"7bd734c0-9b03-463b-adfd-e1a206fd1f9a\") " Oct 30 17:55:10.774734 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:10.774613 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-model-cache\") pod \"7bd734c0-9b03-463b-adfd-e1a206fd1f9a\" (UID: \"7bd734c0-9b03-463b-adfd-e1a206fd1f9a\") " Oct 30 17:55:10.774899 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:10.774808 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-tmp-dir" pod "7bd734c0-9b03-463b-adfd-e1a206fd1f9a" (UID: "7bd734c0-9b03-463b-adfd-e1a206fd1f9a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:55:10.774899 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:10.774841 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-home" pod "7bd734c0-9b03-463b-adfd-e1a206fd1f9a" (UID: "7bd734c0-9b03-463b-adfd-e1a206fd1f9a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:55:10.775031 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:10.775000 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-model-cache" pod "7bd734c0-9b03-463b-adfd-e1a206fd1f9a" (UID: "7bd734c0-9b03-463b-adfd-e1a206fd1f9a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:55:10.775412 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:10.775384 4965 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-tmp-dir\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:55:10.775507 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:10.775417 4965 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-home\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:55:10.775507 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:10.775433 4965 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-model-cache\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:55:10.776712 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:10.776684 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-kube-api-access-mccz9" pod "7bd734c0-9b03-463b-adfd-e1a206fd1f9a" (UID: "7bd734c0-9b03-463b-adfd-e1a206fd1f9a"). InnerVolumeSpecName "kube-api-access-mccz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:55:10.776712 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:10.776690 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-tls-certs" pod "7bd734c0-9b03-463b-adfd-e1a206fd1f9a" (UID: "7bd734c0-9b03-463b-adfd-e1a206fd1f9a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:55:10.776912 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:10.776750 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-dshm" pod "7bd734c0-9b03-463b-adfd-e1a206fd1f9a" (UID: "7bd734c0-9b03-463b-adfd-e1a206fd1f9a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:55:10.861047 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:10.861013 4965 generic.go:356] "Generic (PLEG): container finished" podID="7bd734c0-9b03-463b-adfd-e1a206fd1f9a" containerID="c42c009df56ffbc9b72a033d306165886a91f90bd6ebf97edba95768b120f4c6" exitCode=0 Oct 30 17:55:10.861234 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:10.861074 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" Oct 30 17:55:10.861234 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:10.861108 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" event={"ID":"7bd734c0-9b03-463b-adfd-e1a206fd1f9a","Type":"ContainerDied","Data":"c42c009df56ffbc9b72a033d306165886a91f90bd6ebf97edba95768b120f4c6"} Oct 30 17:55:10.861234 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:10.861160 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx" event={"ID":"7bd734c0-9b03-463b-adfd-e1a206fd1f9a","Type":"ContainerDied","Data":"6613d5dcdb18b5d932764119b23d4e8084892ab0672b29e04500e0db3692e4c0"} Oct 30 17:55:10.861234 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:10.861184 4965 scope.go:122] "RemoveContainer" containerID="c42c009df56ffbc9b72a033d306165886a91f90bd6ebf97edba95768b120f4c6" Oct 30 17:55:10.872823 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:10.872803 4965 scope.go:122] "RemoveContainer" containerID="c42c009df56ffbc9b72a033d306165886a91f90bd6ebf97edba95768b120f4c6" Oct 30 17:55:10.873096 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:55:10.873073 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c42c009df56ffbc9b72a033d306165886a91f90bd6ebf97edba95768b120f4c6\": container with ID starting with c42c009df56ffbc9b72a033d306165886a91f90bd6ebf97edba95768b120f4c6 not found: ID does not exist" containerID="c42c009df56ffbc9b72a033d306165886a91f90bd6ebf97edba95768b120f4c6" Oct 30 17:55:10.873149 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:10.873103 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c42c009df56ffbc9b72a033d306165886a91f90bd6ebf97edba95768b120f4c6"} err="failed to get container status \"c42c009df56ffbc9b72a033d306165886a91f90bd6ebf97edba95768b120f4c6\": rpc error: code = NotFound desc = could not find container \"c42c009df56ffbc9b72a033d306165886a91f90bd6ebf97edba95768b120f4c6\": container with ID starting with c42c009df56ffbc9b72a033d306165886a91f90bd6ebf97edba95768b120f4c6 not found: ID does not exist" Oct 30 17:55:10.876523 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:10.876499 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mccz9\" (UniqueName: \"kubernetes.io/projected/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-kube-api-access-mccz9\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:55:10.876523 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:10.876524 4965 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-tls-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:55:10.876661 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:10.876532 4965 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd734c0-9b03-463b-adfd-e1a206fd1f9a-dshm\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:55:10.914084 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:10.913989 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx"] Oct 30 17:55:10.926137 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:10.918793 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-c31c73/auth-test-model-c31c73-kserve-594654b985-cxdkx"] Oct 30 17:55:12.100292 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:12.100260 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bd734c0-9b03-463b-adfd-e1a206fd1f9a" path="/var/lib/kubelet/pods/7bd734c0-9b03-463b-adfd-e1a206fd1f9a/volumes" Oct 30 17:55:13.890082 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:13.890053 4965 generic.go:356] "Generic (PLEG): container finished" podID="50d223e1-883f-454f-9c29-348129b7d676" containerID="2ca97e7cfca2c2246fab3e6871885e92a10dffa92c8fae7a0e6576dec6495b0b" exitCode=0 Oct 30 17:55:13.890536 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:13.890128 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" event={"ID":"50d223e1-883f-454f-9c29-348129b7d676","Type":"ContainerDied","Data":"2ca97e7cfca2c2246fab3e6871885e92a10dffa92c8fae7a0e6576dec6495b0b"} Oct 30 17:55:13.890536 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:13.890183 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" event={"ID":"50d223e1-883f-454f-9c29-348129b7d676","Type":"ContainerDied","Data":"2ed6a417688766ef132e949a5bb0e61458a32d12b1f302b9948a02fd70c29c66"} Oct 30 17:55:13.890536 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:13.890199 4965 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ed6a417688766ef132e949a5bb0e61458a32d12b1f302b9948a02fd70c29c66" Oct 30 17:55:13.890536 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:13.890213 4965 scope.go:122] "RemoveContainer" containerID="0d0c0f6ce3e971f2cab20611549e4304e190a972dcf33697fd4ecc0fc7b165df" Oct 30 17:55:13.902641 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:13.902624 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:55:14.023076 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:14.023042 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50d223e1-883f-454f-9c29-348129b7d676-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50d223e1-883f-454f-9c29-348129b7d676-istiod-ca-cert\") pod \"50d223e1-883f-454f-9c29-348129b7d676\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " Oct 30 17:55:14.023276 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:14.023113 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-istio-envoy\") pod \"50d223e1-883f-454f-9c29-348129b7d676\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " Oct 30 17:55:14.023276 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:14.023146 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-credential-socket\") pod \"50d223e1-883f-454f-9c29-348129b7d676\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " Oct 30 17:55:14.023276 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:14.023182 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-workload-socket\") pod \"50d223e1-883f-454f-9c29-348129b7d676\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " Oct 30 17:55:14.023276 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:14.023247 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-workload-certs\") pod \"50d223e1-883f-454f-9c29-348129b7d676\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " Oct 30 17:55:14.023505 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:14.023356 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50d223e1-883f-454f-9c29-348129b7d676-istio-token\" (UniqueName: \"kubernetes.io/projected/50d223e1-883f-454f-9c29-348129b7d676-istio-token\") pod \"50d223e1-883f-454f-9c29-348129b7d676\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " Oct 30 17:55:14.023505 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:14.023393 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50d223e1-883f-454f-9c29-348129b7d676-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50d223e1-883f-454f-9c29-348129b7d676-istio-podinfo\") pod \"50d223e1-883f-454f-9c29-348129b7d676\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " Oct 30 17:55:14.023505 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:14.023445 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50d223e1-883f-454f-9c29-348129b7d676-kube-api-access-4np2d\" (UniqueName: \"kubernetes.io/projected/50d223e1-883f-454f-9c29-348129b7d676-kube-api-access-4np2d\") pod \"50d223e1-883f-454f-9c29-348129b7d676\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " Oct 30 17:55:14.023656 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:14.023570 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-istio-data\") pod \"50d223e1-883f-454f-9c29-348129b7d676\" (UID: \"50d223e1-883f-454f-9c29-348129b7d676\") " Oct 30 17:55:14.023735 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:14.023704 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50d223e1-883f-454f-9c29-348129b7d676-istiod-ca-cert" pod "50d223e1-883f-454f-9c29-348129b7d676" (UID: "50d223e1-883f-454f-9c29-348129b7d676"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:55:14.023829 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:14.023805 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-workload-certs" pod "50d223e1-883f-454f-9c29-348129b7d676" (UID: "50d223e1-883f-454f-9c29-348129b7d676"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:55:14.024224 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:14.024197 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-istio-data" pod "50d223e1-883f-454f-9c29-348129b7d676" (UID: "50d223e1-883f-454f-9c29-348129b7d676"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:55:14.024354 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:14.024341 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50d223e1-883f-454f-9c29-348129b7d676-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:55:14.024413 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:14.024366 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:55:14.024413 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:14.024380 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:55:14.024413 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:14.024336 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-workload-socket" pod "50d223e1-883f-454f-9c29-348129b7d676" (UID: "50d223e1-883f-454f-9c29-348129b7d676"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:55:14.024413 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:14.024385 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-credential-socket" pod "50d223e1-883f-454f-9c29-348129b7d676" (UID: "50d223e1-883f-454f-9c29-348129b7d676"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:55:14.026634 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:14.026608 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-istio-envoy" pod "50d223e1-883f-454f-9c29-348129b7d676" (UID: "50d223e1-883f-454f-9c29-348129b7d676"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:55:14.026735 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:14.026655 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50d223e1-883f-454f-9c29-348129b7d676-istio-token" pod "50d223e1-883f-454f-9c29-348129b7d676" (UID: "50d223e1-883f-454f-9c29-348129b7d676"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:55:14.027017 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:14.026992 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50d223e1-883f-454f-9c29-348129b7d676-kube-api-access-4np2d" pod "50d223e1-883f-454f-9c29-348129b7d676" (UID: "50d223e1-883f-454f-9c29-348129b7d676"). InnerVolumeSpecName "kube-api-access-4np2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:55:14.027150 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:14.027130 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50d223e1-883f-454f-9c29-348129b7d676-istio-podinfo" pod "50d223e1-883f-454f-9c29-348129b7d676" (UID: "50d223e1-883f-454f-9c29-348129b7d676"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:55:14.125911 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:14.125816 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:55:14.125911 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:14.125849 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:55:14.125911 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:14.125879 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50d223e1-883f-454f-9c29-348129b7d676-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:55:14.125911 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:14.125895 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50d223e1-883f-454f-9c29-348129b7d676-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:55:14.125911 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:14.125909 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50d223e1-883f-454f-9c29-348129b7d676-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:55:14.126212 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:14.125922 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4np2d\" (UniqueName: \"kubernetes.io/projected/50d223e1-883f-454f-9c29-348129b7d676-kube-api-access-4np2d\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:55:14.899057 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:14.899026 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz" Oct 30 17:55:14.999271 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:14.999235 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz"] Oct 30 17:55:15.006936 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:15.006904 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-bcdac0-openshift-default-5d4954598c-m6bhz"] Oct 30 17:55:16.100484 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:16.100448 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50d223e1-883f-454f-9c29-348129b7d676" path="/var/lib/kubelet/pods/50d223e1-883f-454f-9c29-348129b7d676/volumes" Oct 30 17:55:26.114225 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:26.114192 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn"] Oct 30 17:55:26.114728 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:26.114426 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" podUID="731917aa-271d-4a54-a249-07daad43970c" containerName="istio-proxy" containerID="cri-o://a6b699b93042338bae23e1d9a1cdc11e9c67ff16bd3a9799ee5f3ef0a54bd28b" gracePeriod=30 Oct 30 17:55:28.067187 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:28.067141 4965 patch_prober.go:29] interesting pod/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:55:28.067654 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:28.067220 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" podUID="731917aa-271d-4a54-a249-07daad43970c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:55:31.401200 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:31.401171 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:55:31.422987 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:31.422953 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/731917aa-271d-4a54-a249-07daad43970c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/731917aa-271d-4a54-a249-07daad43970c-istiod-ca-cert\") pod \"731917aa-271d-4a54-a249-07daad43970c\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " Oct 30 17:55:31.423127 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:31.423002 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-istio-data\") pod \"731917aa-271d-4a54-a249-07daad43970c\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " Oct 30 17:55:31.423127 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:31.423071 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-istio-envoy\") pod \"731917aa-271d-4a54-a249-07daad43970c\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " Oct 30 17:55:31.423127 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:31.423105 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/731917aa-271d-4a54-a249-07daad43970c-kube-api-access-wf6kw\" (UniqueName: \"kubernetes.io/projected/731917aa-271d-4a54-a249-07daad43970c-kube-api-access-wf6kw\") pod \"731917aa-271d-4a54-a249-07daad43970c\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " Oct 30 17:55:31.423255 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:31.423133 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-credential-socket\") pod \"731917aa-271d-4a54-a249-07daad43970c\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " Oct 30 17:55:31.423338 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:31.423320 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-workload-socket\") pod \"731917aa-271d-4a54-a249-07daad43970c\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " Oct 30 17:55:31.423410 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:31.423352 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-istio-data" pod "731917aa-271d-4a54-a249-07daad43970c" (UID: "731917aa-271d-4a54-a249-07daad43970c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:55:31.423410 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:31.423370 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/731917aa-271d-4a54-a249-07daad43970c-istio-token\" (UniqueName: \"kubernetes.io/projected/731917aa-271d-4a54-a249-07daad43970c-istio-token\") pod \"731917aa-271d-4a54-a249-07daad43970c\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " Oct 30 17:55:31.423410 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:31.423383 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/731917aa-271d-4a54-a249-07daad43970c-istiod-ca-cert" pod "731917aa-271d-4a54-a249-07daad43970c" (UID: "731917aa-271d-4a54-a249-07daad43970c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:55:31.423572 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:31.423425 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/731917aa-271d-4a54-a249-07daad43970c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/731917aa-271d-4a54-a249-07daad43970c-istio-podinfo\") pod \"731917aa-271d-4a54-a249-07daad43970c\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " Oct 30 17:55:31.423572 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:31.423492 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-workload-certs\") pod \"731917aa-271d-4a54-a249-07daad43970c\" (UID: \"731917aa-271d-4a54-a249-07daad43970c\") " Oct 30 17:55:31.423572 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:31.423544 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-credential-socket" pod "731917aa-271d-4a54-a249-07daad43970c" (UID: "731917aa-271d-4a54-a249-07daad43970c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:55:31.423758 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:31.423608 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-workload-socket" pod "731917aa-271d-4a54-a249-07daad43970c" (UID: "731917aa-271d-4a54-a249-07daad43970c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:55:31.423901 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:31.423857 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-workload-certs" pod "731917aa-271d-4a54-a249-07daad43970c" (UID: "731917aa-271d-4a54-a249-07daad43970c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:55:31.424079 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:31.424059 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:55:31.424132 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:31.424087 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:55:31.424132 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:31.424102 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/731917aa-271d-4a54-a249-07daad43970c-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:55:31.424132 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:31.424113 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:55:31.424132 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:31.424125 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:55:31.425520 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:31.425495 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/731917aa-271d-4a54-a249-07daad43970c-kube-api-access-wf6kw" pod "731917aa-271d-4a54-a249-07daad43970c" (UID: "731917aa-271d-4a54-a249-07daad43970c"). InnerVolumeSpecName "kube-api-access-wf6kw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:55:31.425613 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:31.425589 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/731917aa-271d-4a54-a249-07daad43970c-istio-token" pod "731917aa-271d-4a54-a249-07daad43970c" (UID: "731917aa-271d-4a54-a249-07daad43970c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:55:31.425979 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:31.425958 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-istio-envoy" pod "731917aa-271d-4a54-a249-07daad43970c" (UID: "731917aa-271d-4a54-a249-07daad43970c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:55:31.425979 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:31.425965 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/731917aa-271d-4a54-a249-07daad43970c-istio-podinfo" pod "731917aa-271d-4a54-a249-07daad43970c" (UID: "731917aa-271d-4a54-a249-07daad43970c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:55:31.525327 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:31.525293 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/731917aa-271d-4a54-a249-07daad43970c-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:55:31.525327 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:31.525324 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf6kw\" (UniqueName: \"kubernetes.io/projected/731917aa-271d-4a54-a249-07daad43970c-kube-api-access-wf6kw\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:55:31.525327 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:31.525335 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/731917aa-271d-4a54-a249-07daad43970c-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:55:31.525559 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:31.525344 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/731917aa-271d-4a54-a249-07daad43970c-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:55:32.057167 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:32.057135 4965 generic.go:356] "Generic (PLEG): container finished" podID="731917aa-271d-4a54-a249-07daad43970c" containerID="a6b699b93042338bae23e1d9a1cdc11e9c67ff16bd3a9799ee5f3ef0a54bd28b" exitCode=0 Oct 30 17:55:32.057343 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:32.057197 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" event={"ID":"731917aa-271d-4a54-a249-07daad43970c","Type":"ContainerDied","Data":"a6b699b93042338bae23e1d9a1cdc11e9c67ff16bd3a9799ee5f3ef0a54bd28b"} Oct 30 17:55:32.057343 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:32.057233 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" event={"ID":"731917aa-271d-4a54-a249-07daad43970c","Type":"ContainerDied","Data":"eece93b9a66ed6c5557d1b16bc68b03402b2c5a43805c00361778353c7848d0e"} Oct 30 17:55:32.057343 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:32.057252 4965 scope.go:122] "RemoveContainer" containerID="a6b699b93042338bae23e1d9a1cdc11e9c67ff16bd3a9799ee5f3ef0a54bd28b" Oct 30 17:55:32.057450 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:32.057360 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn" Oct 30 17:55:32.070270 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:32.070243 4965 scope.go:122] "RemoveContainer" containerID="a6b699b93042338bae23e1d9a1cdc11e9c67ff16bd3a9799ee5f3ef0a54bd28b" Oct 30 17:55:32.070542 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:55:32.070515 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6b699b93042338bae23e1d9a1cdc11e9c67ff16bd3a9799ee5f3ef0a54bd28b\": container with ID starting with a6b699b93042338bae23e1d9a1cdc11e9c67ff16bd3a9799ee5f3ef0a54bd28b not found: ID does not exist" containerID="a6b699b93042338bae23e1d9a1cdc11e9c67ff16bd3a9799ee5f3ef0a54bd28b" Oct 30 17:55:32.070592 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:32.070553 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6b699b93042338bae23e1d9a1cdc11e9c67ff16bd3a9799ee5f3ef0a54bd28b"} err="failed to get container status \"a6b699b93042338bae23e1d9a1cdc11e9c67ff16bd3a9799ee5f3ef0a54bd28b\": rpc error: code = NotFound desc = could not find container \"a6b699b93042338bae23e1d9a1cdc11e9c67ff16bd3a9799ee5f3ef0a54bd28b\": container with ID starting with a6b699b93042338bae23e1d9a1cdc11e9c67ff16bd3a9799ee5f3ef0a54bd28b not found: ID does not exist" Oct 30 17:55:32.112482 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:32.112403 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn"] Oct 30 17:55:32.124343 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:32.124318 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4a271c-openshift-default-5fc86cc4c9-9xnxn"] Oct 30 17:55:34.100516 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:34.100481 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="731917aa-271d-4a54-a249-07daad43970c" path="/var/lib/kubelet/pods/731917aa-271d-4a54-a249-07daad43970c/volumes" Oct 30 17:55:36.004424 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:55:36.004390 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:55:37.186842 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:37.186812 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c596f1-778cb5cc94-glfgz"] Oct 30 17:55:37.222244 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:37.222208 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c596f1-7c568c9f95-lgd2d"] Oct 30 17:55:42.187903 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:42.187837 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-c596f1-778cb5cc94-glfgz" podUID="e03a603f-5564-4340-ba6d-395abdf6fbd9" containerName="payload-processing" containerID="cri-o://196b4dacdbac4935f666219b8b8e2ade2c5ff4a6ce26c0f3e0c727c87d9c7cfa" gracePeriod=25 Oct 30 17:55:42.222924 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:42.222883 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c596f1-7c568c9f95-lgd2d" podUID="4cc9400e-90bd-4eb0-bf82-763acc3cf2c7" containerName="payload-pre-processing" containerID="cri-o://f3ec766c443dcb257462c73a1dfa419e8b78c81587f727c7a4c386bc0b294840" gracePeriod=25 Oct 30 17:55:42.535572 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:42.535548 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c596f1-778cb5cc94-glfgz" Oct 30 17:55:42.629126 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:42.629089 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e03a603f-5564-4340-ba6d-395abdf6fbd9-kube-api-access-wps8x\" (UniqueName: \"kubernetes.io/projected/e03a603f-5564-4340-ba6d-395abdf6fbd9-kube-api-access-wps8x\") pod \"e03a603f-5564-4340-ba6d-395abdf6fbd9\" (UID: \"e03a603f-5564-4340-ba6d-395abdf6fbd9\") " Oct 30 17:55:42.629340 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:42.629139 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e03a603f-5564-4340-ba6d-395abdf6fbd9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e03a603f-5564-4340-ba6d-395abdf6fbd9-plugins-config-volume\") pod \"e03a603f-5564-4340-ba6d-395abdf6fbd9\" (UID: \"e03a603f-5564-4340-ba6d-395abdf6fbd9\") " Oct 30 17:55:42.629900 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:42.629840 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e03a603f-5564-4340-ba6d-395abdf6fbd9-plugins-config-volume" pod "e03a603f-5564-4340-ba6d-395abdf6fbd9" (UID: "e03a603f-5564-4340-ba6d-395abdf6fbd9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:55:42.632017 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:42.631984 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e03a603f-5564-4340-ba6d-395abdf6fbd9-kube-api-access-wps8x" pod "e03a603f-5564-4340-ba6d-395abdf6fbd9" (UID: "e03a603f-5564-4340-ba6d-395abdf6fbd9"). InnerVolumeSpecName "kube-api-access-wps8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:55:42.731376 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:42.731341 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wps8x\" (UniqueName: \"kubernetes.io/projected/e03a603f-5564-4340-ba6d-395abdf6fbd9-kube-api-access-wps8x\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:55:42.731376 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:42.731366 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e03a603f-5564-4340-ba6d-395abdf6fbd9-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:55:43.151490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:43.151459 4965 generic.go:356] "Generic (PLEG): container finished" podID="e03a603f-5564-4340-ba6d-395abdf6fbd9" containerID="196b4dacdbac4935f666219b8b8e2ade2c5ff4a6ce26c0f3e0c727c87d9c7cfa" exitCode=0 Oct 30 17:55:43.151671 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:43.151509 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c596f1-778cb5cc94-glfgz" Oct 30 17:55:43.151671 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:43.151542 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c596f1-778cb5cc94-glfgz" event={"ID":"e03a603f-5564-4340-ba6d-395abdf6fbd9","Type":"ContainerDied","Data":"196b4dacdbac4935f666219b8b8e2ade2c5ff4a6ce26c0f3e0c727c87d9c7cfa"} Oct 30 17:55:43.151671 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:43.151611 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c596f1-778cb5cc94-glfgz" event={"ID":"e03a603f-5564-4340-ba6d-395abdf6fbd9","Type":"ContainerDied","Data":"1b388494893398ac725a1406f70a4aa201c57cab40186aa0d43bb3e0b8096de8"} Oct 30 17:55:43.151671 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:43.151633 4965 scope.go:122] "RemoveContainer" containerID="196b4dacdbac4935f666219b8b8e2ade2c5ff4a6ce26c0f3e0c727c87d9c7cfa" Oct 30 17:55:43.162829 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:43.162807 4965 scope.go:122] "RemoveContainer" containerID="196b4dacdbac4935f666219b8b8e2ade2c5ff4a6ce26c0f3e0c727c87d9c7cfa" Oct 30 17:55:43.163124 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:55:43.163096 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"196b4dacdbac4935f666219b8b8e2ade2c5ff4a6ce26c0f3e0c727c87d9c7cfa\": container with ID starting with 196b4dacdbac4935f666219b8b8e2ade2c5ff4a6ce26c0f3e0c727c87d9c7cfa not found: ID does not exist" containerID="196b4dacdbac4935f666219b8b8e2ade2c5ff4a6ce26c0f3e0c727c87d9c7cfa" Oct 30 17:55:43.163202 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:43.163132 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"196b4dacdbac4935f666219b8b8e2ade2c5ff4a6ce26c0f3e0c727c87d9c7cfa"} err="failed to get container status \"196b4dacdbac4935f666219b8b8e2ade2c5ff4a6ce26c0f3e0c727c87d9c7cfa\": rpc error: code = NotFound desc = could not find container \"196b4dacdbac4935f666219b8b8e2ade2c5ff4a6ce26c0f3e0c727c87d9c7cfa\": container with ID starting with 196b4dacdbac4935f666219b8b8e2ade2c5ff4a6ce26c0f3e0c727c87d9c7cfa not found: ID does not exist" Oct 30 17:55:43.204531 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:43.204502 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c596f1-778cb5cc94-glfgz"] Oct 30 17:55:43.216655 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:43.216630 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c596f1-778cb5cc94-glfgz"] Oct 30 17:55:44.100450 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:44.100413 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e03a603f-5564-4340-ba6d-395abdf6fbd9" path="/var/lib/kubelet/pods/e03a603f-5564-4340-ba6d-395abdf6fbd9/volumes" Oct 30 17:55:44.899068 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:44.899032 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c596f1-7c568c9f95-lgd2d" podUID="4cc9400e-90bd-4eb0-bf82-763acc3cf2c7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:55:54.268274 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.268238 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd"] Oct 30 17:55:54.269380 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.269353 4965 cpu_manager.go:414] "containerMap: removing container" podUID="e03a603f-5564-4340-ba6d-395abdf6fbd9" containerName="payload-processing" Oct 30 17:55:54.269496 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.269380 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e03a603f-5564-4340-ba6d-395abdf6fbd9" containerName="payload-processing" Oct 30 17:55:54.269496 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.269417 4965 cpu_manager.go:414] "containerMap: removing container" podUID="50d223e1-883f-454f-9c29-348129b7d676" containerName="istio-proxy" Oct 30 17:55:54.269496 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.269425 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50d223e1-883f-454f-9c29-348129b7d676" containerName="istio-proxy" Oct 30 17:55:54.269496 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.269438 4965 cpu_manager.go:414] "containerMap: removing container" podUID="3af50d6f-9968-40fd-9159-b6f7078023db" containerName="payload-pre-processing" Oct 30 17:55:54.269496 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.269446 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3af50d6f-9968-40fd-9159-b6f7078023db" containerName="payload-pre-processing" Oct 30 17:55:54.269496 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.269473 4965 cpu_manager.go:414] "containerMap: removing container" podUID="731917aa-271d-4a54-a249-07daad43970c" containerName="istio-proxy" Oct 30 17:55:54.269496 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.269481 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="731917aa-271d-4a54-a249-07daad43970c" containerName="istio-proxy" Oct 30 17:55:54.269845 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.269506 4965 cpu_manager.go:414] "containerMap: removing container" podUID="7bd734c0-9b03-463b-adfd-e1a206fd1f9a" containerName="main" Oct 30 17:55:54.269845 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.269513 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bd734c0-9b03-463b-adfd-e1a206fd1f9a" containerName="main" Oct 30 17:55:54.269845 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.269527 4965 cpu_manager.go:414] "containerMap: removing container" podUID="50d223e1-883f-454f-9c29-348129b7d676" containerName="istio-proxy" Oct 30 17:55:54.269845 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.269534 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50d223e1-883f-454f-9c29-348129b7d676" containerName="istio-proxy" Oct 30 17:55:54.269845 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.269687 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="50d223e1-883f-454f-9c29-348129b7d676" containerName="istio-proxy" Oct 30 17:55:54.269845 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.269700 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="3af50d6f-9968-40fd-9159-b6f7078023db" containerName="payload-pre-processing" Oct 30 17:55:54.269845 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.269712 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="731917aa-271d-4a54-a249-07daad43970c" containerName="istio-proxy" Oct 30 17:55:54.269845 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.269724 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bd734c0-9b03-463b-adfd-e1a206fd1f9a" containerName="main" Oct 30 17:55:54.269845 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.269732 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="50d223e1-883f-454f-9c29-348129b7d676" containerName="istio-proxy" Oct 30 17:55:54.269845 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.269751 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="e03a603f-5564-4340-ba6d-395abdf6fbd9" containerName="payload-processing" Oct 30 17:55:54.273733 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.273708 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd" Oct 30 17:55:54.292409 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.292383 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd"] Oct 30 17:55:54.299457 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.299432 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-c31c73\"/\"openshift-service-ca.crt\"" Oct 30 17:55:54.299457 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.299456 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-c31c73\"/\"default-dockercfg-m42t8\"" Oct 30 17:55:54.299664 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.299522 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-c31c73\"/\"kube-root-ca.crt\"" Oct 30 17:55:54.300386 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.300369 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-c31c73\"/\"sub-test-model-c31c73-kserve-self-signed-certs\"" Oct 30 17:55:54.331844 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.331807 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-home\") pod \"sub-test-model-c31c73-kserve-6cdfb75949-vwxkd\" (UID: \"a2f007f6-27a0-4cfb-ba7e-c20f030397f1\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd" Oct 30 17:55:54.331978 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.331945 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-dshm\") pod \"sub-test-model-c31c73-kserve-6cdfb75949-vwxkd\" (UID: \"a2f007f6-27a0-4cfb-ba7e-c20f030397f1\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd" Oct 30 17:55:54.332045 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.332028 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-model-cache\") pod \"sub-test-model-c31c73-kserve-6cdfb75949-vwxkd\" (UID: \"a2f007f6-27a0-4cfb-ba7e-c20f030397f1\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd" Oct 30 17:55:54.332094 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.332067 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtvt5\" (UniqueName: \"kubernetes.io/projected/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-kube-api-access-mtvt5\") pod \"sub-test-model-c31c73-kserve-6cdfb75949-vwxkd\" (UID: \"a2f007f6-27a0-4cfb-ba7e-c20f030397f1\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd" Oct 30 17:55:54.332286 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.332264 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-tmp-dir\") pod \"sub-test-model-c31c73-kserve-6cdfb75949-vwxkd\" (UID: \"a2f007f6-27a0-4cfb-ba7e-c20f030397f1\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd" Oct 30 17:55:54.332334 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.332314 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-tls-certs\") pod \"sub-test-model-c31c73-kserve-6cdfb75949-vwxkd\" (UID: \"a2f007f6-27a0-4cfb-ba7e-c20f030397f1\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd" Oct 30 17:55:54.433603 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.433572 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-home\") pod \"sub-test-model-c31c73-kserve-6cdfb75949-vwxkd\" (UID: \"a2f007f6-27a0-4cfb-ba7e-c20f030397f1\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd" Oct 30 17:55:54.433774 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.433646 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-dshm\") pod \"sub-test-model-c31c73-kserve-6cdfb75949-vwxkd\" (UID: \"a2f007f6-27a0-4cfb-ba7e-c20f030397f1\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd" Oct 30 17:55:54.433774 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.433749 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-model-cache\") pod \"sub-test-model-c31c73-kserve-6cdfb75949-vwxkd\" (UID: \"a2f007f6-27a0-4cfb-ba7e-c20f030397f1\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd" Oct 30 17:55:54.433911 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.433791 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtvt5\" (UniqueName: \"kubernetes.io/projected/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-kube-api-access-mtvt5\") pod \"sub-test-model-c31c73-kserve-6cdfb75949-vwxkd\" (UID: \"a2f007f6-27a0-4cfb-ba7e-c20f030397f1\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd" Oct 30 17:55:54.434064 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.434038 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-tmp-dir\") pod \"sub-test-model-c31c73-kserve-6cdfb75949-vwxkd\" (UID: \"a2f007f6-27a0-4cfb-ba7e-c20f030397f1\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd" Oct 30 17:55:54.434159 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.434096 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-tls-certs\") pod \"sub-test-model-c31c73-kserve-6cdfb75949-vwxkd\" (UID: \"a2f007f6-27a0-4cfb-ba7e-c20f030397f1\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd" Oct 30 17:55:54.434159 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.434044 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-home\") pod \"sub-test-model-c31c73-kserve-6cdfb75949-vwxkd\" (UID: \"a2f007f6-27a0-4cfb-ba7e-c20f030397f1\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd" Oct 30 17:55:54.434526 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.434504 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-model-cache\") pod \"sub-test-model-c31c73-kserve-6cdfb75949-vwxkd\" (UID: \"a2f007f6-27a0-4cfb-ba7e-c20f030397f1\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd" Oct 30 17:55:54.434669 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.434528 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-tmp-dir\") pod \"sub-test-model-c31c73-kserve-6cdfb75949-vwxkd\" (UID: \"a2f007f6-27a0-4cfb-ba7e-c20f030397f1\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd" Oct 30 17:55:54.436442 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.436415 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-dshm\") pod \"sub-test-model-c31c73-kserve-6cdfb75949-vwxkd\" (UID: \"a2f007f6-27a0-4cfb-ba7e-c20f030397f1\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd" Oct 30 17:55:54.437109 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.436923 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-tls-certs\") pod \"sub-test-model-c31c73-kserve-6cdfb75949-vwxkd\" (UID: \"a2f007f6-27a0-4cfb-ba7e-c20f030397f1\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd" Oct 30 17:55:54.457144 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.457119 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtvt5\" (UniqueName: \"kubernetes.io/projected/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-kube-api-access-mtvt5\") pod \"sub-test-model-c31c73-kserve-6cdfb75949-vwxkd\" (UID: \"a2f007f6-27a0-4cfb-ba7e-c20f030397f1\") " pod="ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd" Oct 30 17:55:54.593222 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.592332 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd" Oct 30 17:55:54.899384 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.899308 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c596f1-7c568c9f95-lgd2d" podUID="4cc9400e-90bd-4eb0-bf82-763acc3cf2c7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:55:54.937826 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:54.937796 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd"] Oct 30 17:55:54.939033 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:55:54.939005 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2f007f6_27a0_4cfb_ba7e_c20f030397f1.slice/crio-342f9a1cb7fe04aefe1081caecabc795859c354aed91b4cfc64325a7097d87dd WatchSource:0}: Error finding container 342f9a1cb7fe04aefe1081caecabc795859c354aed91b4cfc64325a7097d87dd: Status 404 returned error can't find the container with id 342f9a1cb7fe04aefe1081caecabc795859c354aed91b4cfc64325a7097d87dd Oct 30 17:55:55.134843 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.134805 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962"] Oct 30 17:55:55.135114 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.135084 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962" podUID="3a0b9f7e-5cae-4e6d-9737-6aef893fda01" containerName="maas-api" containerID="cri-o://040992c2496f62a774fb072d58125b021815a4db92023699f49739b4753d8fc2" gracePeriod=30 Oct 30 17:55:55.280700 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.280668 4965 generic.go:356] "Generic (PLEG): container finished" podID="3a0b9f7e-5cae-4e6d-9737-6aef893fda01" containerID="040992c2496f62a774fb072d58125b021815a4db92023699f49739b4753d8fc2" exitCode=0 Oct 30 17:55:55.281177 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.280754 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962" event={"ID":"3a0b9f7e-5cae-4e6d-9737-6aef893fda01","Type":"ContainerDied","Data":"040992c2496f62a774fb072d58125b021815a4db92023699f49739b4753d8fc2"} Oct 30 17:55:55.282674 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.282641 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd" event={"ID":"a2f007f6-27a0-4cfb-ba7e-c20f030397f1","Type":"ContainerStarted","Data":"06fa90304cb5bdb5c5fc146fa6f7e1a8a91180a71ecb10209dd7a70df739e6f1"} Oct 30 17:55:55.282836 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.282689 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd" event={"ID":"a2f007f6-27a0-4cfb-ba7e-c20f030397f1","Type":"ContainerStarted","Data":"342f9a1cb7fe04aefe1081caecabc795859c354aed91b4cfc64325a7097d87dd"} Oct 30 17:55:55.336408 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.336344 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd" podStartSLOduration=1.3363231230000001 podStartE2EDuration="1.336323123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:55:55.33173621 +0000 UTC m=+2613.383836144" watchObservedRunningTime="2026-10-30 17:55:55.336323123 +0000 UTC m=+2613.388423055" Oct 30 17:55:55.390576 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.390544 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn"] Oct 30 17:55:55.396640 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.396619 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:55:55.416380 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.416353 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn"] Oct 30 17:55:55.417454 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.417437 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962" Oct 30 17:55:55.447927 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.441907 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-a2a6cb-openshift-default-dockercfg-qs2lw\"" Oct 30 17:55:55.450717 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.450581 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a0b9f7e-5cae-4e6d-9737-6aef893fda01-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a0b9f7e-5cae-4e6d-9737-6aef893fda01-maas-api-tls\") pod \"3a0b9f7e-5cae-4e6d-9737-6aef893fda01\" (UID: \"3a0b9f7e-5cae-4e6d-9737-6aef893fda01\") " Oct 30 17:55:55.450852 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.450731 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a0b9f7e-5cae-4e6d-9737-6aef893fda01-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a0b9f7e-5cae-4e6d-9737-6aef893fda01-openshift-service-ca\") pod \"3a0b9f7e-5cae-4e6d-9737-6aef893fda01\" (UID: \"3a0b9f7e-5cae-4e6d-9737-6aef893fda01\") " Oct 30 17:55:55.450852 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.450793 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a0b9f7e-5cae-4e6d-9737-6aef893fda01-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a0b9f7e-5cae-4e6d-9737-6aef893fda01-metrics-certs\") pod \"3a0b9f7e-5cae-4e6d-9737-6aef893fda01\" (UID: \"3a0b9f7e-5cae-4e6d-9737-6aef893fda01\") " Oct 30 17:55:55.451114 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.450883 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a0b9f7e-5cae-4e6d-9737-6aef893fda01-kube-api-access-qtjn4\" (UniqueName: \"kubernetes.io/projected/3a0b9f7e-5cae-4e6d-9737-6aef893fda01-kube-api-access-qtjn4\") pod \"3a0b9f7e-5cae-4e6d-9737-6aef893fda01\" (UID: \"3a0b9f7e-5cae-4e6d-9737-6aef893fda01\") " Oct 30 17:55:55.451242 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.451151 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-istio-envoy\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:55:55.451383 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.451273 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-workload-socket\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:55:55.451436 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.451379 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdjsn\" (UniqueName: \"kubernetes.io/projected/ef413002-b509-4658-a123-a5b4b0715d58-kube-api-access-fdjsn\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:55:55.451504 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.451477 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef413002-b509-4658-a123-a5b4b0715d58-istiod-ca-cert\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:55:55.451581 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.451564 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef413002-b509-4658-a123-a5b4b0715d58-istio-podinfo\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:55:55.451646 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.451611 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-workload-certs\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:55:55.451698 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.451672 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-istio-data\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:55:55.451752 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.451716 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-credential-socket\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:55:55.451790 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.451753 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef413002-b509-4658-a123-a5b4b0715d58-istio-token\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:55:55.454031 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.454000 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a0b9f7e-5cae-4e6d-9737-6aef893fda01-openshift-service-ca" pod "3a0b9f7e-5cae-4e6d-9737-6aef893fda01" (UID: "3a0b9f7e-5cae-4e6d-9737-6aef893fda01"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:55:55.455238 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.455195 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a0b9f7e-5cae-4e6d-9737-6aef893fda01-maas-api-tls" pod "3a0b9f7e-5cae-4e6d-9737-6aef893fda01" (UID: "3a0b9f7e-5cae-4e6d-9737-6aef893fda01"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:55:55.455238 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.455219 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a0b9f7e-5cae-4e6d-9737-6aef893fda01-metrics-certs" pod "3a0b9f7e-5cae-4e6d-9737-6aef893fda01" (UID: "3a0b9f7e-5cae-4e6d-9737-6aef893fda01"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:55:55.457471 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.457438 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a0b9f7e-5cae-4e6d-9737-6aef893fda01-kube-api-access-qtjn4" pod "3a0b9f7e-5cae-4e6d-9737-6aef893fda01" (UID: "3a0b9f7e-5cae-4e6d-9737-6aef893fda01"). InnerVolumeSpecName "kube-api-access-qtjn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:55:55.554212 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.554130 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-istio-data\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:55:55.554212 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.554178 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-credential-socket\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:55:55.554212 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.554201 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef413002-b509-4658-a123-a5b4b0715d58-istio-token\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:55:55.554439 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.554283 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-istio-envoy\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:55:55.554439 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.554354 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-workload-socket\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:55:55.554520 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.554444 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdjsn\" (UniqueName: \"kubernetes.io/projected/ef413002-b509-4658-a123-a5b4b0715d58-kube-api-access-fdjsn\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:55:55.554520 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.554482 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef413002-b509-4658-a123-a5b4b0715d58-istiod-ca-cert\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:55:55.554626 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.554564 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef413002-b509-4658-a123-a5b4b0715d58-istio-podinfo\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:55:55.554626 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.554586 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-istio-data\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:55:55.554626 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.554619 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-workload-certs\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:55:55.554793 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.554631 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-credential-socket\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:55:55.554793 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.554751 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-workload-socket\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:55:55.554959 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.554793 4965 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a0b9f7e-5cae-4e6d-9737-6aef893fda01-openshift-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:55:55.554959 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.554808 4965 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a0b9f7e-5cae-4e6d-9737-6aef893fda01-metrics-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:55:55.554959 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.554833 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtjn4\" (UniqueName: \"kubernetes.io/projected/3a0b9f7e-5cae-4e6d-9737-6aef893fda01-kube-api-access-qtjn4\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:55:55.554959 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.554845 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a0b9f7e-5cae-4e6d-9737-6aef893fda01-maas-api-tls\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:55:55.555157 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.555071 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-workload-certs\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:55:55.555227 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.555206 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef413002-b509-4658-a123-a5b4b0715d58-istiod-ca-cert\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:55:55.556800 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.556778 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-istio-envoy\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:55:55.557322 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.557299 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef413002-b509-4658-a123-a5b4b0715d58-istio-podinfo\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:55:55.577394 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.577368 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef413002-b509-4658-a123-a5b4b0715d58-istio-token\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:55:55.577595 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.577573 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdjsn\" (UniqueName: \"kubernetes.io/projected/ef413002-b509-4658-a123-a5b4b0715d58-kube-api-access-fdjsn\") pod \"e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:55:55.593351 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.593326 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd" Oct 30 17:55:55.714611 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:55.713821 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:55:56.093077 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:55:56.093048 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef413002_b509_4658_a123_a5b4b0715d58.slice/crio-f1caf9382008cb6b8afaf0920e6a953e3848e38cd439a52689b0dbfa05772fed WatchSource:0}: Error finding container f1caf9382008cb6b8afaf0920e6a953e3848e38cd439a52689b0dbfa05772fed: Status 404 returned error can't find the container with id f1caf9382008cb6b8afaf0920e6a953e3848e38cd439a52689b0dbfa05772fed Oct 30 17:55:56.095761 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:56.095724 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:55:56.095881 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:56.095793 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:55:56.095881 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:56.095823 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:55:56.101882 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:56.101844 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn"] Oct 30 17:55:56.292469 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:56.292436 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" event={"ID":"ef413002-b509-4658-a123-a5b4b0715d58","Type":"ContainerStarted","Data":"7e36fc6874820207e79e41512526c66c0a1ce2c376bb3908150814c6a42a9084"} Oct 30 17:55:56.292998 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:56.292496 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" event={"ID":"ef413002-b509-4658-a123-a5b4b0715d58","Type":"ContainerStarted","Data":"f1caf9382008cb6b8afaf0920e6a953e3848e38cd439a52689b0dbfa05772fed"} Oct 30 17:55:56.293823 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:56.293804 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962" Oct 30 17:55:56.293982 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:56.293844 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962" event={"ID":"3a0b9f7e-5cae-4e6d-9737-6aef893fda01","Type":"ContainerDied","Data":"1413d4236cc0674d582496db35b123519f6fedce02bda231a65f316b58dd3e4f"} Oct 30 17:55:56.293982 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:56.293912 4965 scope.go:122] "RemoveContainer" containerID="040992c2496f62a774fb072d58125b021815a4db92023699f49739b4753d8fc2" Oct 30 17:55:56.337733 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:56.337228 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" podStartSLOduration=1.337213128 podStartE2EDuration="1.337213128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:55:56.336950166 +0000 UTC m=+2614.389050107" watchObservedRunningTime="2026-10-30 17:55:56.337213128 +0000 UTC m=+2614.389313061" Oct 30 17:55:56.375158 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:56.375090 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962"] Oct 30 17:55:56.379910 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:56.379872 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c596f1-6f9867dd48-c7962"] Oct 30 17:55:56.714941 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:56.714833 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:55:57.068943 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:57.068297 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:55:57.717427 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:57.717390 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 17:55:57.717801 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:57.717449 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" podUID="ef413002-b509-4658-a123-a5b4b0715d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 17:55:58.129123 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:58.128355 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a0b9f7e-5cae-4e6d-9737-6aef893fda01" path="/var/lib/kubelet/pods/3a0b9f7e-5cae-4e6d-9737-6aef893fda01/volumes" Oct 30 17:55:58.715238 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:58.715197 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 17:55:58.715407 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:58.715250 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" podUID="ef413002-b509-4658-a123-a5b4b0715d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 17:55:59.714887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:59.714744 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 17:55:59.714887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:55:59.714804 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" podUID="ef413002-b509-4658-a123-a5b4b0715d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 17:56:00.715287 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:00.715250 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 17:56:00.715694 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:00.715301 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" podUID="ef413002-b509-4658-a123-a5b4b0715d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 17:56:01.715363 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:01.715326 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 17:56:01.715762 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:01.715395 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" podUID="ef413002-b509-4658-a123-a5b4b0715d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 17:56:02.391927 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:02.391855 4965 generic.go:356] "Generic (PLEG): container finished" podID="4cc9400e-90bd-4eb0-bf82-763acc3cf2c7" containerID="f3ec766c443dcb257462c73a1dfa419e8b78c81587f727c7a4c386bc0b294840" exitCode=0 Oct 30 17:56:02.392250 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:02.391925 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c596f1-7c568c9f95-lgd2d" event={"ID":"4cc9400e-90bd-4eb0-bf82-763acc3cf2c7","Type":"ContainerDied","Data":"f3ec766c443dcb257462c73a1dfa419e8b78c81587f727c7a4c386bc0b294840"} Oct 30 17:56:02.553589 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:02.553561 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c596f1-7c568c9f95-lgd2d" Oct 30 17:56:02.686882 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:02.686831 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cc9400e-90bd-4eb0-bf82-763acc3cf2c7-kube-api-access-nwtmg\" (UniqueName: \"kubernetes.io/projected/4cc9400e-90bd-4eb0-bf82-763acc3cf2c7-kube-api-access-nwtmg\") pod \"4cc9400e-90bd-4eb0-bf82-763acc3cf2c7\" (UID: \"4cc9400e-90bd-4eb0-bf82-763acc3cf2c7\") " Oct 30 17:56:02.687088 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:02.686926 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cc9400e-90bd-4eb0-bf82-763acc3cf2c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cc9400e-90bd-4eb0-bf82-763acc3cf2c7-plugins-config-volume\") pod \"4cc9400e-90bd-4eb0-bf82-763acc3cf2c7\" (UID: \"4cc9400e-90bd-4eb0-bf82-763acc3cf2c7\") " Oct 30 17:56:02.687275 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:02.687250 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cc9400e-90bd-4eb0-bf82-763acc3cf2c7-plugins-config-volume" pod "4cc9400e-90bd-4eb0-bf82-763acc3cf2c7" (UID: "4cc9400e-90bd-4eb0-bf82-763acc3cf2c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:56:02.687531 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:02.687515 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cc9400e-90bd-4eb0-bf82-763acc3cf2c7-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:56:02.688894 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:02.688845 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc9400e-90bd-4eb0-bf82-763acc3cf2c7-kube-api-access-nwtmg" pod "4cc9400e-90bd-4eb0-bf82-763acc3cf2c7" (UID: "4cc9400e-90bd-4eb0-bf82-763acc3cf2c7"). InnerVolumeSpecName "kube-api-access-nwtmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:56:02.714743 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:02.714715 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 17:56:02.714916 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:02.714766 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" podUID="ef413002-b509-4658-a123-a5b4b0715d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 17:56:02.789528 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:02.789439 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwtmg\" (UniqueName: \"kubernetes.io/projected/4cc9400e-90bd-4eb0-bf82-763acc3cf2c7-kube-api-access-nwtmg\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:56:03.403917 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:03.403882 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c596f1-7c568c9f95-lgd2d" Oct 30 17:56:03.403917 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:03.403897 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c596f1-7c568c9f95-lgd2d" event={"ID":"4cc9400e-90bd-4eb0-bf82-763acc3cf2c7","Type":"ContainerDied","Data":"098359e68d721edc7afb2dd62a078abd2f96dc6cc3afcd7dc867b491c7e4cd78"} Oct 30 17:56:03.404189 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:03.403969 4965 scope.go:122] "RemoveContainer" containerID="f3ec766c443dcb257462c73a1dfa419e8b78c81587f727c7a4c386bc0b294840" Oct 30 17:56:03.426771 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:03.424776 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g"] Oct 30 17:56:03.426771 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:03.425033 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" podUID="b8cbab33-ea35-4e2e-a82d-74e1f47c5b13" containerName="istio-proxy" containerID="cri-o://dd71ba8c3685642fd40121ff243662611cfedc852dd4644b540f39ab6e9bd42f" gracePeriod=30 Oct 30 17:56:03.583885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:03.581947 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c596f1-7c568c9f95-lgd2d"] Oct 30 17:56:03.642164 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:03.642119 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c596f1-7c568c9f95-lgd2d"] Oct 30 17:56:03.719542 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:03.719462 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:56:03.719858 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:03.719835 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:56:03.720779 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:03.720753 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 17:56:04.100450 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.100416 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cc9400e-90bd-4eb0-bf82-763acc3cf2c7" path="/var/lib/kubelet/pods/4cc9400e-90bd-4eb0-bf82-763acc3cf2c7/volumes" Oct 30 17:56:04.593247 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.593217 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd" Oct 30 17:56:04.606771 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.606748 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd" Oct 30 17:56:04.741191 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.741158 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs"] Oct 30 17:56:04.742098 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.742076 4965 cpu_manager.go:414] "containerMap: removing container" podUID="4cc9400e-90bd-4eb0-bf82-763acc3cf2c7" containerName="payload-pre-processing" Oct 30 17:56:04.742098 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.742096 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cc9400e-90bd-4eb0-bf82-763acc3cf2c7" containerName="payload-pre-processing" Oct 30 17:56:04.742315 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.742109 4965 cpu_manager.go:414] "containerMap: removing container" podUID="3a0b9f7e-5cae-4e6d-9737-6aef893fda01" containerName="maas-api" Oct 30 17:56:04.742315 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.742114 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a0b9f7e-5cae-4e6d-9737-6aef893fda01" containerName="maas-api" Oct 30 17:56:04.742315 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.742244 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cc9400e-90bd-4eb0-bf82-763acc3cf2c7" containerName="payload-pre-processing" Oct 30 17:56:04.742315 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.742263 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a0b9f7e-5cae-4e6d-9737-6aef893fda01" containerName="maas-api" Oct 30 17:56:04.747622 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.747602 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:04.766572 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.766548 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-90b48a-openshift-default-dockercfg-gwwvs\"" Oct 30 17:56:04.775813 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.775785 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs"] Oct 30 17:56:04.872132 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.872043 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istio-data\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:04.872132 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.872097 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-workload-socket\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:04.872375 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.872189 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istiod-ca-cert\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:04.872375 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.872248 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istio-token\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:04.872375 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.872342 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istio-envoy\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:04.872527 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.872448 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-credential-socket\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:04.872527 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.872517 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istio-podinfo\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:04.872653 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.872594 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-workload-certs\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:04.872694 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.872668 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k2dh\" (UniqueName: \"kubernetes.io/projected/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-kube-api-access-2k2dh\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:04.997891 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.973740 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istio-token\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:04.997891 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.973806 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istio-envoy\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:04.997891 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.973903 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-credential-socket\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:04.997891 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.973956 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istio-podinfo\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:04.997891 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.974078 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-workload-certs\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:04.997891 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.974132 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k2dh\" (UniqueName: \"kubernetes.io/projected/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-kube-api-access-2k2dh\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:04.997891 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.974220 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istio-data\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:04.997891 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.974265 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-workload-socket\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:04.997891 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.974307 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istiod-ca-cert\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:04.997891 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.975191 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istiod-ca-cert\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:04.997891 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.978230 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-workload-certs\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:04.997891 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.978498 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-credential-socket\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:04.997891 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.979173 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istio-data\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:04.997891 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:04.979675 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-workload-socket\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:05.003894 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:05.002732 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istio-podinfo\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:05.028356 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:05.028325 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istio-envoy\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:05.040884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:05.033426 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k2dh\" (UniqueName: \"kubernetes.io/projected/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-kube-api-access-2k2dh\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:05.040884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:05.034005 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istio-token\") pod \"e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:05.076096 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:05.076062 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:05.431506 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:05.431427 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd" Oct 30 17:56:05.440743 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:05.440684 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs"] Oct 30 17:56:05.444229 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:56:05.444204 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2edd6ff6_320a_42d4_83ef_5173fcbf7b92.slice/crio-8fdb075ba5a10f8cf7afc4007442b1f0285f8f047cd9253ebd364a284567c977 WatchSource:0}: Error finding container 8fdb075ba5a10f8cf7afc4007442b1f0285f8f047cd9253ebd364a284567c977: Status 404 returned error can't find the container with id 8fdb075ba5a10f8cf7afc4007442b1f0285f8f047cd9253ebd364a284567c977 Oct 30 17:56:05.447368 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:05.447335 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:56:05.447468 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:05.447412 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:56:05.447468 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:05.447457 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:56:05.998723 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:56:05.998684 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:56:06.430117 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:06.430076 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" event={"ID":"2edd6ff6-320a-42d4-83ef-5173fcbf7b92","Type":"ContainerStarted","Data":"0a653659716682526a9f79db0df47f6c6b0b5aa5119f56d2ad6ada9fd7fda922"} Oct 30 17:56:06.430313 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:06.430143 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" event={"ID":"2edd6ff6-320a-42d4-83ef-5173fcbf7b92","Type":"ContainerStarted","Data":"8fdb075ba5a10f8cf7afc4007442b1f0285f8f047cd9253ebd364a284567c977"} Oct 30 17:56:06.476434 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:06.476381 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" podStartSLOduration=2.476362299 podStartE2EDuration="2.476362299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:06.474602156 +0000 UTC m=+2624.526702087" watchObservedRunningTime="2026-10-30 17:56:06.476362299 +0000 UTC m=+2624.528462229" Oct 30 17:56:07.078585 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:07.078548 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:07.088416 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:07.088384 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:07.438466 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:07.438371 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:07.439276 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:07.439247 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:56:08.257639 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:08.257386 4965 patch_prober.go:29] interesting pod/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:56:08.258150 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:08.257559 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" podUID="b8cbab33-ea35-4e2e-a82d-74e1f47c5b13" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:56:08.718072 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:08.718046 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:56:08.796277 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:08.796189 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-workload-certs\") pod \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " Oct 30 17:56:08.796445 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:08.796325 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-kube-api-access-lmjpq\" (UniqueName: \"kubernetes.io/projected/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-kube-api-access-lmjpq\") pod \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " Oct 30 17:56:08.796445 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:08.796420 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istio-data\") pod \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " Oct 30 17:56:08.796578 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:08.796457 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-workload-certs" pod "b8cbab33-ea35-4e2e-a82d-74e1f47c5b13" (UID: "b8cbab33-ea35-4e2e-a82d-74e1f47c5b13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:56:08.796578 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:08.796475 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istio-envoy\") pod \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " Oct 30 17:56:08.796578 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:08.796538 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-credential-socket\") pod \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " Oct 30 17:56:08.796578 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:08.796571 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istiod-ca-cert\") pod \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " Oct 30 17:56:08.796806 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:08.796686 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-workload-socket\") pod \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " Oct 30 17:56:08.796806 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:08.796685 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istio-data" pod "b8cbab33-ea35-4e2e-a82d-74e1f47c5b13" (UID: "b8cbab33-ea35-4e2e-a82d-74e1f47c5b13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:56:08.796806 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:08.796742 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istio-podinfo\") pod \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " Oct 30 17:56:08.796999 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:08.796830 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-credential-socket" pod "b8cbab33-ea35-4e2e-a82d-74e1f47c5b13" (UID: "b8cbab33-ea35-4e2e-a82d-74e1f47c5b13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:56:08.796999 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:08.796843 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istio-token\" (UniqueName: \"kubernetes.io/projected/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istio-token\") pod \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\" (UID: \"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13\") " Oct 30 17:56:08.796999 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:08.796942 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-workload-socket" pod "b8cbab33-ea35-4e2e-a82d-74e1f47c5b13" (UID: "b8cbab33-ea35-4e2e-a82d-74e1f47c5b13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:56:08.797184 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:08.797045 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istiod-ca-cert" pod "b8cbab33-ea35-4e2e-a82d-74e1f47c5b13" (UID: "b8cbab33-ea35-4e2e-a82d-74e1f47c5b13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:56:08.797537 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:08.797508 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:56:08.797661 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:08.797540 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:56:08.797661 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:08.797553 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:56:08.797661 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:08.797566 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:56:08.797661 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:08.797581 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:56:08.798888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:08.798838 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-kube-api-access-lmjpq" pod "b8cbab33-ea35-4e2e-a82d-74e1f47c5b13" (UID: "b8cbab33-ea35-4e2e-a82d-74e1f47c5b13"). InnerVolumeSpecName "kube-api-access-lmjpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:56:08.799455 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:08.799428 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istio-token" pod "b8cbab33-ea35-4e2e-a82d-74e1f47c5b13" (UID: "b8cbab33-ea35-4e2e-a82d-74e1f47c5b13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:56:08.799529 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:08.799427 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istio-podinfo" pod "b8cbab33-ea35-4e2e-a82d-74e1f47c5b13" (UID: "b8cbab33-ea35-4e2e-a82d-74e1f47c5b13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:56:08.800003 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:08.799979 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istio-envoy" pod "b8cbab33-ea35-4e2e-a82d-74e1f47c5b13" (UID: "b8cbab33-ea35-4e2e-a82d-74e1f47c5b13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:56:08.899109 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:08.899069 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:56:08.899109 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:08.899103 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:56:08.899314 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:08.899117 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmjpq\" (UniqueName: \"kubernetes.io/projected/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-kube-api-access-lmjpq\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:56:08.899314 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:08.899130 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:56:09.455680 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:09.455646 4965 generic.go:356] "Generic (PLEG): container finished" podID="b8cbab33-ea35-4e2e-a82d-74e1f47c5b13" containerID="dd71ba8c3685642fd40121ff243662611cfedc852dd4644b540f39ab6e9bd42f" exitCode=0 Oct 30 17:56:09.456172 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:09.455693 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" Oct 30 17:56:09.456172 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:09.455729 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" event={"ID":"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13","Type":"ContainerDied","Data":"dd71ba8c3685642fd40121ff243662611cfedc852dd4644b540f39ab6e9bd42f"} Oct 30 17:56:09.456172 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:09.455792 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g" event={"ID":"b8cbab33-ea35-4e2e-a82d-74e1f47c5b13","Type":"ContainerDied","Data":"4114eaee609667222ad80169da93f19825fb1f20a33dbf87942d43f4d647514a"} Oct 30 17:56:09.456172 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:09.455832 4965 scope.go:122] "RemoveContainer" containerID="dd71ba8c3685642fd40121ff243662611cfedc852dd4644b540f39ab6e9bd42f" Oct 30 17:56:09.467760 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:09.467738 4965 scope.go:122] "RemoveContainer" containerID="ff71ab324ee976af17b669afc245f2fc029c7e1dda2cde3f69189ac352495334" Oct 30 17:56:09.478038 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:09.478021 4965 scope.go:122] "RemoveContainer" containerID="dd71ba8c3685642fd40121ff243662611cfedc852dd4644b540f39ab6e9bd42f" Oct 30 17:56:09.478340 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:56:09.478319 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd71ba8c3685642fd40121ff243662611cfedc852dd4644b540f39ab6e9bd42f\": container with ID starting with dd71ba8c3685642fd40121ff243662611cfedc852dd4644b540f39ab6e9bd42f not found: ID does not exist" containerID="dd71ba8c3685642fd40121ff243662611cfedc852dd4644b540f39ab6e9bd42f" Oct 30 17:56:09.478401 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:09.478352 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd71ba8c3685642fd40121ff243662611cfedc852dd4644b540f39ab6e9bd42f"} err="failed to get container status \"dd71ba8c3685642fd40121ff243662611cfedc852dd4644b540f39ab6e9bd42f\": rpc error: code = NotFound desc = could not find container \"dd71ba8c3685642fd40121ff243662611cfedc852dd4644b540f39ab6e9bd42f\": container with ID starting with dd71ba8c3685642fd40121ff243662611cfedc852dd4644b540f39ab6e9bd42f not found: ID does not exist" Oct 30 17:56:09.478401 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:09.478374 4965 scope.go:122] "RemoveContainer" containerID="ff71ab324ee976af17b669afc245f2fc029c7e1dda2cde3f69189ac352495334" Oct 30 17:56:09.478632 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:56:09.478613 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff71ab324ee976af17b669afc245f2fc029c7e1dda2cde3f69189ac352495334\": container with ID starting with ff71ab324ee976af17b669afc245f2fc029c7e1dda2cde3f69189ac352495334 not found: ID does not exist" containerID="ff71ab324ee976af17b669afc245f2fc029c7e1dda2cde3f69189ac352495334" Oct 30 17:56:09.478673 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:09.478638 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff71ab324ee976af17b669afc245f2fc029c7e1dda2cde3f69189ac352495334"} err="failed to get container status \"ff71ab324ee976af17b669afc245f2fc029c7e1dda2cde3f69189ac352495334\": rpc error: code = NotFound desc = could not find container \"ff71ab324ee976af17b669afc245f2fc029c7e1dda2cde3f69189ac352495334\": container with ID starting with ff71ab324ee976af17b669afc245f2fc029c7e1dda2cde3f69189ac352495334 not found: ID does not exist" Oct 30 17:56:09.520828 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:09.520800 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g"] Oct 30 17:56:09.532645 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:09.532620 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c596f1-openshift-default-69cb4f46cc-f965g"] Oct 30 17:56:10.100654 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:10.100620 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8cbab33-ea35-4e2e-a82d-74e1f47c5b13" path="/var/lib/kubelet/pods/b8cbab33-ea35-4e2e-a82d-74e1f47c5b13/volumes" Oct 30 17:56:10.843810 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:10.843777 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-90b48a-745c46d66c-v42dh"] Oct 30 17:56:10.844488 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:10.844473 4965 cpu_manager.go:414] "containerMap: removing container" podUID="b8cbab33-ea35-4e2e-a82d-74e1f47c5b13" containerName="istio-proxy" Oct 30 17:56:10.844540 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:10.844489 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8cbab33-ea35-4e2e-a82d-74e1f47c5b13" containerName="istio-proxy" Oct 30 17:56:10.844540 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:10.844502 4965 cpu_manager.go:414] "containerMap: removing container" podUID="b8cbab33-ea35-4e2e-a82d-74e1f47c5b13" containerName="istio-proxy" Oct 30 17:56:10.844540 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:10.844508 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8cbab33-ea35-4e2e-a82d-74e1f47c5b13" containerName="istio-proxy" Oct 30 17:56:10.844661 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:10.844652 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8cbab33-ea35-4e2e-a82d-74e1f47c5b13" containerName="istio-proxy" Oct 30 17:56:10.844701 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:10.844664 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8cbab33-ea35-4e2e-a82d-74e1f47c5b13" containerName="istio-proxy" Oct 30 17:56:10.849686 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:10.849666 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-90b48a-745c46d66c-v42dh" Oct 30 17:56:10.862059 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:10.862036 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-90b48a-7d6d7f978b-v22m7"] Oct 30 17:56:10.866338 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:10.866322 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-90b48a-7d6d7f978b-v22m7" Oct 30 17:56:10.878832 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:10.878809 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-90b48a\"" Oct 30 17:56:10.879845 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:10.879825 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-90b48a-745c46d66c-v42dh"] Oct 30 17:56:10.886415 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:10.886390 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-90b48a-dockercfg-zbj29\"" Oct 30 17:56:10.886531 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:10.886500 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-90b48a-7d6d7f978b-v22m7"] Oct 30 17:56:10.928195 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:10.928160 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mtpw\" (UniqueName: \"kubernetes.io/projected/844e6c1f-9e6d-4890-93e0-4216c37bc999-kube-api-access-7mtpw\") pod \"payload-pre-processing-e2e-shared-a-w3-90b48a-745c46d66c-v42dh\" (UID: \"844e6c1f-9e6d-4890-93e0-4216c37bc999\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-90b48a-745c46d66c-v42dh" Oct 30 17:56:10.928372 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:10.928353 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4296e00-981c-4a22-8f06-d8fc997c359e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-90b48a-7d6d7f978b-v22m7\" (UID: \"f4296e00-981c-4a22-8f06-d8fc997c359e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-90b48a-7d6d7f978b-v22m7" Oct 30 17:56:10.928496 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:10.928479 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844e6c1f-9e6d-4890-93e0-4216c37bc999-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-90b48a-745c46d66c-v42dh\" (UID: \"844e6c1f-9e6d-4890-93e0-4216c37bc999\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-90b48a-745c46d66c-v42dh" Oct 30 17:56:10.928533 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:10.928518 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxpq5\" (UniqueName: \"kubernetes.io/projected/f4296e00-981c-4a22-8f06-d8fc997c359e-kube-api-access-xxpq5\") pod \"payload-processing-e2e-shared-a-w3-90b48a-7d6d7f978b-v22m7\" (UID: \"f4296e00-981c-4a22-8f06-d8fc997c359e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-90b48a-7d6d7f978b-v22m7" Oct 30 17:56:11.054885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:11.032737 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844e6c1f-9e6d-4890-93e0-4216c37bc999-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-90b48a-745c46d66c-v42dh\" (UID: \"844e6c1f-9e6d-4890-93e0-4216c37bc999\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-90b48a-745c46d66c-v42dh" Oct 30 17:56:11.054885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:11.032796 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxpq5\" (UniqueName: \"kubernetes.io/projected/f4296e00-981c-4a22-8f06-d8fc997c359e-kube-api-access-xxpq5\") pod \"payload-processing-e2e-shared-a-w3-90b48a-7d6d7f978b-v22m7\" (UID: \"f4296e00-981c-4a22-8f06-d8fc997c359e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-90b48a-7d6d7f978b-v22m7" Oct 30 17:56:11.054885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:11.032988 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mtpw\" (UniqueName: \"kubernetes.io/projected/844e6c1f-9e6d-4890-93e0-4216c37bc999-kube-api-access-7mtpw\") pod \"payload-pre-processing-e2e-shared-a-w3-90b48a-745c46d66c-v42dh\" (UID: \"844e6c1f-9e6d-4890-93e0-4216c37bc999\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-90b48a-745c46d66c-v42dh" Oct 30 17:56:11.054885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:11.033125 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4296e00-981c-4a22-8f06-d8fc997c359e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-90b48a-7d6d7f978b-v22m7\" (UID: \"f4296e00-981c-4a22-8f06-d8fc997c359e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-90b48a-7d6d7f978b-v22m7" Oct 30 17:56:11.054885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:11.034188 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4296e00-981c-4a22-8f06-d8fc997c359e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-90b48a-7d6d7f978b-v22m7\" (UID: \"f4296e00-981c-4a22-8f06-d8fc997c359e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-90b48a-7d6d7f978b-v22m7" Oct 30 17:56:11.055923 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:11.055540 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844e6c1f-9e6d-4890-93e0-4216c37bc999-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-90b48a-745c46d66c-v42dh\" (UID: \"844e6c1f-9e6d-4890-93e0-4216c37bc999\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-90b48a-745c46d66c-v42dh" Oct 30 17:56:11.080264 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:11.080232 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mtpw\" (UniqueName: \"kubernetes.io/projected/844e6c1f-9e6d-4890-93e0-4216c37bc999-kube-api-access-7mtpw\") pod \"payload-pre-processing-e2e-shared-a-w3-90b48a-745c46d66c-v42dh\" (UID: \"844e6c1f-9e6d-4890-93e0-4216c37bc999\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-90b48a-745c46d66c-v42dh" Oct 30 17:56:11.109016 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:11.098683 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxpq5\" (UniqueName: \"kubernetes.io/projected/f4296e00-981c-4a22-8f06-d8fc997c359e-kube-api-access-xxpq5\") pod \"payload-processing-e2e-shared-a-w3-90b48a-7d6d7f978b-v22m7\" (UID: \"f4296e00-981c-4a22-8f06-d8fc997c359e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-90b48a-7d6d7f978b-v22m7" Oct 30 17:56:11.165259 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:11.165226 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-90b48a-745c46d66c-v42dh" Oct 30 17:56:11.183724 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:11.183700 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-90b48a-7d6d7f978b-v22m7" Oct 30 17:56:11.477709 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:56:11.477679 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844e6c1f_9e6d_4890_93e0_4216c37bc999.slice/crio-9396e9ebd4663fb2e3f56798efa1419ffdc0c519fa14ce421f23d6d536cdb81a WatchSource:0}: Error finding container 9396e9ebd4663fb2e3f56798efa1419ffdc0c519fa14ce421f23d6d536cdb81a: Status 404 returned error can't find the container with id 9396e9ebd4663fb2e3f56798efa1419ffdc0c519fa14ce421f23d6d536cdb81a Oct 30 17:56:11.480146 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:11.480121 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-90b48a-745c46d66c-v42dh"] Oct 30 17:56:11.502536 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:56:11.502508 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4296e00_981c_4a22_8f06_d8fc997c359e.slice/crio-48ac9ef20b3bb43dd88a433b6833330d498b2fb6f71e963a61cd884e370d47df WatchSource:0}: Error finding container 48ac9ef20b3bb43dd88a433b6833330d498b2fb6f71e963a61cd884e370d47df: Status 404 returned error can't find the container with id 48ac9ef20b3bb43dd88a433b6833330d498b2fb6f71e963a61cd884e370d47df Oct 30 17:56:11.503732 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:11.503706 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-90b48a-7d6d7f978b-v22m7"] Oct 30 17:56:12.481533 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:12.481495 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-90b48a-7d6d7f978b-v22m7" event={"ID":"f4296e00-981c-4a22-8f06-d8fc997c359e","Type":"ContainerStarted","Data":"5a5e9c989bb0b699c9086b3feec00d0823c66778c5c94fd30614142db2a2ca7e"} Oct 30 17:56:12.482013 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:12.481561 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-90b48a-7d6d7f978b-v22m7" event={"ID":"f4296e00-981c-4a22-8f06-d8fc997c359e","Type":"ContainerStarted","Data":"48ac9ef20b3bb43dd88a433b6833330d498b2fb6f71e963a61cd884e370d47df"} Oct 30 17:56:12.482013 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:12.481595 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-90b48a-7d6d7f978b-v22m7" Oct 30 17:56:12.483391 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:12.483366 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-90b48a-745c46d66c-v42dh" event={"ID":"844e6c1f-9e6d-4890-93e0-4216c37bc999","Type":"ContainerStarted","Data":"ca15ce206f4bd06d526942ad08dd24bcbda6296f023787645ef8cdb0dfd676c4"} Oct 30 17:56:12.483517 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:12.483408 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-90b48a-745c46d66c-v42dh" event={"ID":"844e6c1f-9e6d-4890-93e0-4216c37bc999","Type":"ContainerStarted","Data":"9396e9ebd4663fb2e3f56798efa1419ffdc0c519fa14ce421f23d6d536cdb81a"} Oct 30 17:56:12.483517 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:12.483430 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-90b48a-745c46d66c-v42dh" Oct 30 17:56:12.525285 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:12.525241 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-90b48a-7d6d7f978b-v22m7" podStartSLOduration=2.525227359 podStartE2EDuration="2.525227359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:12.523814928 +0000 UTC m=+2630.575914879" watchObservedRunningTime="2026-10-30 17:56:12.525227359 +0000 UTC m=+2630.577327349" Oct 30 17:56:12.567952 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:12.567900 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-90b48a-745c46d66c-v42dh" podStartSLOduration=2.56788333 podStartE2EDuration="2.56788333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:12.564200897 +0000 UTC m=+2630.616300829" watchObservedRunningTime="2026-10-30 17:56:12.56788333 +0000 UTC m=+2630.619983262" Oct 30 17:56:23.493873 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:23.493831 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-90b48a-7d6d7f978b-v22m7" Oct 30 17:56:23.494388 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:23.493920 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-90b48a-745c46d66c-v42dh" Oct 30 17:56:26.201252 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:26.201223 4965 scope.go:122] "RemoveContainer" containerID="2ca97e7cfca2c2246fab3e6871885e92a10dffa92c8fae7a0e6576dec6495b0b" Oct 30 17:56:36.000718 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:56:36.000631 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:56:43.961662 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:43.961630 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw"] Oct 30 17:56:43.966755 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:43.966738 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw" Oct 30 17:56:43.982373 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:43.982343 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-90b48a-serving-cert\"" Oct 30 17:56:43.999487 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:43.999452 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw"] Oct 30 17:56:44.013140 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:44.013105 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb5ef536-ab9a-423e-bea3-dac81549b81c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw\" (UID: \"cb5ef536-ab9a-423e-bea3-dac81549b81c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw" Oct 30 17:56:44.013319 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:44.013184 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb5ef536-ab9a-423e-bea3-dac81549b81c-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw\" (UID: \"cb5ef536-ab9a-423e-bea3-dac81549b81c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw" Oct 30 17:56:44.013319 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:44.013216 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9szn\" (UniqueName: \"kubernetes.io/projected/cb5ef536-ab9a-423e-bea3-dac81549b81c-kube-api-access-t9szn\") pod \"maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw\" (UID: \"cb5ef536-ab9a-423e-bea3-dac81549b81c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw" Oct 30 17:56:44.013319 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:44.013308 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb5ef536-ab9a-423e-bea3-dac81549b81c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw\" (UID: \"cb5ef536-ab9a-423e-bea3-dac81549b81c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw" Oct 30 17:56:44.115106 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:44.115072 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb5ef536-ab9a-423e-bea3-dac81549b81c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw\" (UID: \"cb5ef536-ab9a-423e-bea3-dac81549b81c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw" Oct 30 17:56:44.115307 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:44.115130 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb5ef536-ab9a-423e-bea3-dac81549b81c-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw\" (UID: \"cb5ef536-ab9a-423e-bea3-dac81549b81c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw" Oct 30 17:56:44.115307 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:44.115151 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9szn\" (UniqueName: \"kubernetes.io/projected/cb5ef536-ab9a-423e-bea3-dac81549b81c-kube-api-access-t9szn\") pod \"maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw\" (UID: \"cb5ef536-ab9a-423e-bea3-dac81549b81c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw" Oct 30 17:56:44.115307 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:44.115215 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb5ef536-ab9a-423e-bea3-dac81549b81c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw\" (UID: \"cb5ef536-ab9a-423e-bea3-dac81549b81c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw" Oct 30 17:56:44.115938 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:44.115909 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb5ef536-ab9a-423e-bea3-dac81549b81c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw\" (UID: \"cb5ef536-ab9a-423e-bea3-dac81549b81c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw" Oct 30 17:56:44.117613 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:44.117592 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb5ef536-ab9a-423e-bea3-dac81549b81c-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw\" (UID: \"cb5ef536-ab9a-423e-bea3-dac81549b81c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw" Oct 30 17:56:44.117909 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:44.117857 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb5ef536-ab9a-423e-bea3-dac81549b81c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw\" (UID: \"cb5ef536-ab9a-423e-bea3-dac81549b81c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw" Oct 30 17:56:44.138275 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:44.138246 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9szn\" (UniqueName: \"kubernetes.io/projected/cb5ef536-ab9a-423e-bea3-dac81549b81c-kube-api-access-t9szn\") pod \"maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw\" (UID: \"cb5ef536-ab9a-423e-bea3-dac81549b81c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw" Oct 30 17:56:44.281721 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:44.281615 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw" Oct 30 17:56:44.636670 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:44.636635 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw"] Oct 30 17:56:44.637286 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:56:44.637257 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb5ef536_ab9a_423e_bea3_dac81549b81c.slice/crio-438d4d0d281bedd35410d69bbb9c5906478c36f004d7cee7a9d9a26ca2139d97 WatchSource:0}: Error finding container 438d4d0d281bedd35410d69bbb9c5906478c36f004d7cee7a9d9a26ca2139d97: Status 404 returned error can't find the container with id 438d4d0d281bedd35410d69bbb9c5906478c36f004d7cee7a9d9a26ca2139d97 Oct 30 17:56:44.809235 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:44.809196 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw" event={"ID":"cb5ef536-ab9a-423e-bea3-dac81549b81c","Type":"ContainerStarted","Data":"438d4d0d281bedd35410d69bbb9c5906478c36f004d7cee7a9d9a26ca2139d97"} Oct 30 17:56:45.818283 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:45.818237 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw" event={"ID":"cb5ef536-ab9a-423e-bea3-dac81549b81c","Type":"ContainerStarted","Data":"2fdcde90525f32ca6599745d898ab226bf785f5f8be6b47c3b253cd42f68f383"} Oct 30 17:56:45.818784 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:45.818320 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw" Oct 30 17:56:45.864366 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:45.864318 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw" podStartSLOduration=2.414322638 podStartE2EDuration="2.864303718s" totalImagesPullingTime="449.98108ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:45.861426557 +0000 UTC m=+2663.913526497" watchObservedRunningTime="2026-10-30 17:56:45.864303718 +0000 UTC m=+2663.916403646" Oct 30 17:56:50.717563 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:50.717526 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-21870f-66f8f9975b-56cqf"] Oct 30 17:56:50.722290 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:50.722271 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-66f8f9975b-56cqf" Oct 30 17:56:50.740414 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:50.740383 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-21870f-66f8f9975b-56cqf"] Oct 30 17:56:50.747413 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:50.747380 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-21870f-dockercfg-qf2g6\"" Oct 30 17:56:50.747584 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:50.747419 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-21870f\"" Oct 30 17:56:50.759955 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:50.759925 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-65cf8b4b8c-d4nqw"] Oct 30 17:56:50.764597 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:50.764574 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-65cf8b4b8c-d4nqw" Oct 30 17:56:50.792645 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:50.792617 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-65cf8b4b8c-d4nqw"] Oct 30 17:56:50.866882 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:50.855910 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28fdef67-ee11-47ab-a45a-d7c62873fbea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-21870f-65cf8b4b8c-d4nqw\" (UID: \"28fdef67-ee11-47ab-a45a-d7c62873fbea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-65cf8b4b8c-d4nqw" Oct 30 17:56:50.866882 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:50.856027 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgdbb\" (UniqueName: \"kubernetes.io/projected/a8a3ced7-cbc1-457b-8004-a0175518e383-kube-api-access-vgdbb\") pod \"payload-processing-e2e-ipp-a-21870f-66f8f9975b-56cqf\" (UID: \"a8a3ced7-cbc1-457b-8004-a0175518e383\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-66f8f9975b-56cqf" Oct 30 17:56:50.866882 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:50.856172 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8a3ced7-cbc1-457b-8004-a0175518e383-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-21870f-66f8f9975b-56cqf\" (UID: \"a8a3ced7-cbc1-457b-8004-a0175518e383\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-66f8f9975b-56cqf" Oct 30 17:56:50.866882 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:50.856214 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr8m7\" (UniqueName: \"kubernetes.io/projected/28fdef67-ee11-47ab-a45a-d7c62873fbea-kube-api-access-hr8m7\") pod \"payload-pre-processing-e2e-ipp-a-21870f-65cf8b4b8c-d4nqw\" (UID: \"28fdef67-ee11-47ab-a45a-d7c62873fbea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-65cf8b4b8c-d4nqw" Oct 30 17:56:50.993461 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:50.971724 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28fdef67-ee11-47ab-a45a-d7c62873fbea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-21870f-65cf8b4b8c-d4nqw\" (UID: \"28fdef67-ee11-47ab-a45a-d7c62873fbea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-65cf8b4b8c-d4nqw" Oct 30 17:56:50.993461 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:50.971831 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgdbb\" (UniqueName: \"kubernetes.io/projected/a8a3ced7-cbc1-457b-8004-a0175518e383-kube-api-access-vgdbb\") pod \"payload-processing-e2e-ipp-a-21870f-66f8f9975b-56cqf\" (UID: \"a8a3ced7-cbc1-457b-8004-a0175518e383\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-66f8f9975b-56cqf" Oct 30 17:56:50.993461 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:50.971967 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8a3ced7-cbc1-457b-8004-a0175518e383-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-21870f-66f8f9975b-56cqf\" (UID: \"a8a3ced7-cbc1-457b-8004-a0175518e383\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-66f8f9975b-56cqf" Oct 30 17:56:50.993461 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:50.972023 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr8m7\" (UniqueName: \"kubernetes.io/projected/28fdef67-ee11-47ab-a45a-d7c62873fbea-kube-api-access-hr8m7\") pod \"payload-pre-processing-e2e-ipp-a-21870f-65cf8b4b8c-d4nqw\" (UID: \"28fdef67-ee11-47ab-a45a-d7c62873fbea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-65cf8b4b8c-d4nqw" Oct 30 17:56:50.993461 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:50.973977 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28fdef67-ee11-47ab-a45a-d7c62873fbea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-21870f-65cf8b4b8c-d4nqw\" (UID: \"28fdef67-ee11-47ab-a45a-d7c62873fbea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-65cf8b4b8c-d4nqw" Oct 30 17:56:50.993461 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:50.975207 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8a3ced7-cbc1-457b-8004-a0175518e383-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-21870f-66f8f9975b-56cqf\" (UID: \"a8a3ced7-cbc1-457b-8004-a0175518e383\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-66f8f9975b-56cqf" Oct 30 17:56:51.004746 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:51.004719 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgdbb\" (UniqueName: \"kubernetes.io/projected/a8a3ced7-cbc1-457b-8004-a0175518e383-kube-api-access-vgdbb\") pod \"payload-processing-e2e-ipp-a-21870f-66f8f9975b-56cqf\" (UID: \"a8a3ced7-cbc1-457b-8004-a0175518e383\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-66f8f9975b-56cqf" Oct 30 17:56:51.006246 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:51.006223 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr8m7\" (UniqueName: \"kubernetes.io/projected/28fdef67-ee11-47ab-a45a-d7c62873fbea-kube-api-access-hr8m7\") pod \"payload-pre-processing-e2e-ipp-a-21870f-65cf8b4b8c-d4nqw\" (UID: \"28fdef67-ee11-47ab-a45a-d7c62873fbea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-65cf8b4b8c-d4nqw" Oct 30 17:56:51.037115 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:51.037083 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-66f8f9975b-56cqf" Oct 30 17:56:51.089924 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:51.089887 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-65cf8b4b8c-d4nqw" Oct 30 17:56:51.346741 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:51.346717 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-21870f-66f8f9975b-56cqf"] Oct 30 17:56:51.347194 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:56:51.347170 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a3ced7_cbc1_457b_8004_a0175518e383.slice/crio-dbb32172f987b31b0c23edbd6341e115da3c60a86b4ae33042c24d9297c7434b WatchSource:0}: Error finding container dbb32172f987b31b0c23edbd6341e115da3c60a86b4ae33042c24d9297c7434b: Status 404 returned error can't find the container with id dbb32172f987b31b0c23edbd6341e115da3c60a86b4ae33042c24d9297c7434b Oct 30 17:56:51.416582 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:56:51.416556 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28fdef67_ee11_47ab_a45a_d7c62873fbea.slice/crio-88018755012b94b92c35f8731bc23e7bb5161fa6ba8710074827f5cc29d686d9 WatchSource:0}: Error finding container 88018755012b94b92c35f8731bc23e7bb5161fa6ba8710074827f5cc29d686d9: Status 404 returned error can't find the container with id 88018755012b94b92c35f8731bc23e7bb5161fa6ba8710074827f5cc29d686d9 Oct 30 17:56:51.417219 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:51.417198 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-65cf8b4b8c-d4nqw"] Oct 30 17:56:51.836216 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:51.836180 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw" Oct 30 17:56:52.018081 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:52.018039 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-66f8f9975b-56cqf" event={"ID":"a8a3ced7-cbc1-457b-8004-a0175518e383","Type":"ContainerStarted","Data":"a7aad68be56286b60413a401585f2f2976b6fa7b5bad25f2a4bfbfbcfc95b038"} Oct 30 17:56:52.018274 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:52.018109 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-66f8f9975b-56cqf" Oct 30 17:56:52.018274 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:52.018133 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-66f8f9975b-56cqf" event={"ID":"a8a3ced7-cbc1-457b-8004-a0175518e383","Type":"ContainerStarted","Data":"dbb32172f987b31b0c23edbd6341e115da3c60a86b4ae33042c24d9297c7434b"} Oct 30 17:56:52.019852 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:52.019686 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-65cf8b4b8c-d4nqw" event={"ID":"28fdef67-ee11-47ab-a45a-d7c62873fbea","Type":"ContainerStarted","Data":"b4806c51972ac28f8b69f107a9d68beb59398caf0c967700955a82c476db377e"} Oct 30 17:56:52.019852 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:52.019735 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-65cf8b4b8c-d4nqw" event={"ID":"28fdef67-ee11-47ab-a45a-d7c62873fbea","Type":"ContainerStarted","Data":"88018755012b94b92c35f8731bc23e7bb5161fa6ba8710074827f5cc29d686d9"} Oct 30 17:56:52.019852 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:52.019819 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-65cf8b4b8c-d4nqw" Oct 30 17:56:52.061687 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:52.061635 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-66f8f9975b-56cqf" podStartSLOduration=2.061621077 podStartE2EDuration="2.061621077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:52.05742257 +0000 UTC m=+2670.109522501" watchObservedRunningTime="2026-10-30 17:56:52.061621077 +0000 UTC m=+2670.113721041" Oct 30 17:56:52.098745 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:56:52.098633 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-65cf8b4b8c-d4nqw" podStartSLOduration=2.098616876 podStartE2EDuration="2.098616876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:52.097368574 +0000 UTC m=+2670.149468505" watchObservedRunningTime="2026-10-30 17:56:52.098616876 +0000 UTC m=+2670.150716808" Oct 30 17:57:03.031811 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:03.031781 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-66f8f9975b-56cqf" Oct 30 17:57:03.032360 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:03.031846 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-65cf8b4b8c-d4nqw" Oct 30 17:57:06.002170 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:57:06.002123 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:57:09.955888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:09.954239 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:57:22.321426 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:22.321391 4965 generic.go:356] "Generic (PLEG): container finished" podID="2edd6ff6-320a-42d4-83ef-5173fcbf7b92" containerID="0a653659716682526a9f79db0df47f6c6b0b5aa5119f56d2ad6ada9fd7fda922" exitCode=0 Oct 30 17:57:22.321966 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:22.321453 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" event={"ID":"2edd6ff6-320a-42d4-83ef-5173fcbf7b92","Type":"ContainerDied","Data":"0a653659716682526a9f79db0df47f6c6b0b5aa5119f56d2ad6ada9fd7fda922"} Oct 30 17:57:22.321966 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:22.321941 4965 scope.go:122] "RemoveContainer" containerID="0a653659716682526a9f79db0df47f6c6b0b5aa5119f56d2ad6ada9fd7fda922" Oct 30 17:57:22.323686 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:22.323652 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:57:22.323780 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:22.323726 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:57:22.323780 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:22.323760 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:57:22.466954 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:22.462316 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:57:22.712157 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:22.712094 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 17:57:22.728417 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:22.728387 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 17:57:23.331124 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:23.331092 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" event={"ID":"2edd6ff6-320a-42d4-83ef-5173fcbf7b92","Type":"ContainerStarted","Data":"1baca4c4e7f67345f25459682c044f9aec02944e4e8c4be5efa1a5b8b3510621"} Oct 30 17:57:24.084977 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:24.079825 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:57:24.085192 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:24.084978 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 17:57:24.085192 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:24.085041 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" podUID="2edd6ff6-320a-42d4-83ef-5173fcbf7b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 17:57:25.079287 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:25.079244 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 17:57:25.079911 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:25.079876 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" podUID="2edd6ff6-320a-42d4-83ef-5173fcbf7b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 17:57:26.083022 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:26.081272 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 17:57:26.083022 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:26.081335 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" podUID="2edd6ff6-320a-42d4-83ef-5173fcbf7b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 17:57:27.082392 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:27.082350 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 17:57:27.082623 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:27.082414 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" podUID="2edd6ff6-320a-42d4-83ef-5173fcbf7b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 17:57:28.080721 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:28.080276 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 17:57:28.080721 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:28.080353 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" podUID="2edd6ff6-320a-42d4-83ef-5173fcbf7b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 17:57:29.083882 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:29.083830 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:57:29.084296 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:29.084274 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:57:29.085124 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:29.085103 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 17:57:35.998208 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:57:35.998168 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:57:36.286729 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:36.286642 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg"] Oct 30 17:57:36.292670 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:36.292647 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:36.318525 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:36.318476 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg"] Oct 30 17:57:36.324063 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:36.324031 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-016f1d-openshift-default-dockercfg-jvhbm\"" Oct 30 17:57:36.462692 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:36.462651 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istio-podinfo\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:36.462891 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:36.462707 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-workload-certs\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:36.462954 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:36.462850 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istio-token\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:36.462954 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:36.462943 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-credential-socket\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:36.463039 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:36.462972 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istio-envoy\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:36.463039 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:36.462987 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nldq2\" (UniqueName: \"kubernetes.io/projected/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-kube-api-access-nldq2\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:36.463126 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:36.463058 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-workload-socket\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:36.463126 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:36.463094 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istiod-ca-cert\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:36.463210 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:36.463200 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istio-data\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:36.578960 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:36.578925 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istio-podinfo\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:36.579157 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:36.578993 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-workload-certs\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:36.579157 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:36.579089 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istio-token\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:36.579157 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:36.579141 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-credential-socket\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:36.579338 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:36.579182 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istio-envoy\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:36.579338 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:36.579208 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nldq2\" (UniqueName: \"kubernetes.io/projected/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-kube-api-access-nldq2\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:36.579338 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:36.579246 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-workload-socket\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:36.579338 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:36.579291 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istiod-ca-cert\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:36.579540 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:36.579424 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istio-data\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:36.590884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:36.580021 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istio-data\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:36.590884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:36.580302 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-credential-socket\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:36.590884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:36.581564 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istiod-ca-cert\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:36.590884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:36.582785 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-workload-certs\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:36.590884 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:36.582733 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-workload-socket\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:36.634609 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:36.634539 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istio-envoy\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:36.634927 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:36.634898 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istio-token\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:36.635512 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:36.635487 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istio-podinfo\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:36.638096 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:36.638049 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nldq2\" (UniqueName: \"kubernetes.io/projected/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-kube-api-access-nldq2\") pod \"e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:36.907647 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:36.907547 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:37.275302 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:37.275263 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg"] Oct 30 17:57:37.286030 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:37.285986 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:57:37.286190 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:37.286067 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:57:37.286190 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:37.286101 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:57:37.459357 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:37.459323 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" event={"ID":"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9","Type":"ContainerStarted","Data":"d3eb8d14fd0c7a2c882f5077197e73d9111a4619bbf7aa9152dc83749d177814"} Oct 30 17:57:37.459540 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:37.459381 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" event={"ID":"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9","Type":"ContainerStarted","Data":"812d4ed6b1b166c075c9aea0ddc17d41f79ccdfcf42b15b1bfa0e7610f5e3bc8"} Oct 30 17:57:37.506699 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:37.506642 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" podStartSLOduration=1.506626858 podStartE2EDuration="1.506626858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:37.503254409 +0000 UTC m=+2715.555354359" watchObservedRunningTime="2026-10-30 17:57:37.506626858 +0000 UTC m=+2715.558726788" Oct 30 17:57:37.908157 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:37.908121 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:38.909966 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:38.909926 4965 patch_prober.go:29] interesting pod/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 17:57:38.910457 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:38.909983 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" podUID="6e3c3d72-4ad1-4368-93e4-5f429a7f80e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 17:57:39.940543 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:39.936112 4965 patch_prober.go:29] interesting pod/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 17:57:39.940543 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:39.936174 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" podUID="6e3c3d72-4ad1-4368-93e4-5f429a7f80e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 17:57:40.908081 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:40.908040 4965 patch_prober.go:29] interesting pod/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 17:57:40.908292 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:40.908100 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" podUID="6e3c3d72-4ad1-4368-93e4-5f429a7f80e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 17:57:41.908361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:41.908324 4965 patch_prober.go:29] interesting pod/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 17:57:41.908788 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:41.908401 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" podUID="6e3c3d72-4ad1-4368-93e4-5f429a7f80e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 17:57:42.688407 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:42.688365 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-738845-867b76d77d-qzjng"] Oct 30 17:57:42.694784 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:42.694757 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-738845-867b76d77d-qzjng"] Oct 30 17:57:42.694953 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:42.694898 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-738845-867b76d77d-qzjng" Oct 30 17:57:42.702412 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:42.702376 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdxfr\" (UniqueName: \"kubernetes.io/projected/8abc8140-8a36-4cc7-80da-43738e049fe0-kube-api-access-bdxfr\") pod \"payload-processing-e2e-shared-b-w3-738845-867b76d77d-qzjng\" (UID: \"8abc8140-8a36-4cc7-80da-43738e049fe0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-738845-867b76d77d-qzjng" Oct 30 17:57:42.702571 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:42.702424 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8abc8140-8a36-4cc7-80da-43738e049fe0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-738845-867b76d77d-qzjng\" (UID: \"8abc8140-8a36-4cc7-80da-43738e049fe0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-738845-867b76d77d-qzjng" Oct 30 17:57:42.718707 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:42.718684 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-738845\"" Oct 30 17:57:42.742576 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:42.742547 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-738845-6c477d4fb7-r4tnv"] Oct 30 17:57:42.747289 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:42.747266 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-738845-6c477d4fb7-r4tnv" Oct 30 17:57:42.786073 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:42.786040 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-738845-dockercfg-2hp5z\"" Oct 30 17:57:42.794909 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:42.792998 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-738845-6c477d4fb7-r4tnv"] Oct 30 17:57:42.842892 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:42.820807 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8smb\" (UniqueName: \"kubernetes.io/projected/dcb3f0ba-021a-4788-ad64-fcc9054fe5a1-kube-api-access-w8smb\") pod \"payload-pre-processing-e2e-shared-b-w3-738845-6c477d4fb7-r4tnv\" (UID: \"dcb3f0ba-021a-4788-ad64-fcc9054fe5a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-738845-6c477d4fb7-r4tnv" Oct 30 17:57:42.843389 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:42.843359 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdxfr\" (UniqueName: \"kubernetes.io/projected/8abc8140-8a36-4cc7-80da-43738e049fe0-kube-api-access-bdxfr\") pod \"payload-processing-e2e-shared-b-w3-738845-867b76d77d-qzjng\" (UID: \"8abc8140-8a36-4cc7-80da-43738e049fe0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-738845-867b76d77d-qzjng" Oct 30 17:57:42.844798 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:42.844766 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8abc8140-8a36-4cc7-80da-43738e049fe0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-738845-867b76d77d-qzjng\" (UID: \"8abc8140-8a36-4cc7-80da-43738e049fe0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-738845-867b76d77d-qzjng" Oct 30 17:57:42.845086 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:42.845024 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8abc8140-8a36-4cc7-80da-43738e049fe0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-738845-867b76d77d-qzjng\" (UID: \"8abc8140-8a36-4cc7-80da-43738e049fe0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-738845-867b76d77d-qzjng" Oct 30 17:57:42.845229 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:42.845211 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb3f0ba-021a-4788-ad64-fcc9054fe5a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-738845-6c477d4fb7-r4tnv\" (UID: \"dcb3f0ba-021a-4788-ad64-fcc9054fe5a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-738845-6c477d4fb7-r4tnv" Oct 30 17:57:42.960953 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:42.949268 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8smb\" (UniqueName: \"kubernetes.io/projected/dcb3f0ba-021a-4788-ad64-fcc9054fe5a1-kube-api-access-w8smb\") pod \"payload-pre-processing-e2e-shared-b-w3-738845-6c477d4fb7-r4tnv\" (UID: \"dcb3f0ba-021a-4788-ad64-fcc9054fe5a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-738845-6c477d4fb7-r4tnv" Oct 30 17:57:42.990940 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:42.976217 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb3f0ba-021a-4788-ad64-fcc9054fe5a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-738845-6c477d4fb7-r4tnv\" (UID: \"dcb3f0ba-021a-4788-ad64-fcc9054fe5a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-738845-6c477d4fb7-r4tnv" Oct 30 17:57:43.006460 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:42.990794 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb3f0ba-021a-4788-ad64-fcc9054fe5a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-738845-6c477d4fb7-r4tnv\" (UID: \"dcb3f0ba-021a-4788-ad64-fcc9054fe5a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-738845-6c477d4fb7-r4tnv" Oct 30 17:57:43.012749 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:42.962568 4965 patch_prober.go:29] interesting pod/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 17:57:43.037948 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:43.023076 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" podUID="6e3c3d72-4ad1-4368-93e4-5f429a7f80e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 17:57:43.055033 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:43.044247 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdxfr\" (UniqueName: \"kubernetes.io/projected/8abc8140-8a36-4cc7-80da-43738e049fe0-kube-api-access-bdxfr\") pod \"payload-processing-e2e-shared-b-w3-738845-867b76d77d-qzjng\" (UID: \"8abc8140-8a36-4cc7-80da-43738e049fe0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-738845-867b76d77d-qzjng" Oct 30 17:57:43.055033 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:43.049374 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8smb\" (UniqueName: \"kubernetes.io/projected/dcb3f0ba-021a-4788-ad64-fcc9054fe5a1-kube-api-access-w8smb\") pod \"payload-pre-processing-e2e-shared-b-w3-738845-6c477d4fb7-r4tnv\" (UID: \"dcb3f0ba-021a-4788-ad64-fcc9054fe5a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-738845-6c477d4fb7-r4tnv" Oct 30 17:57:43.098242 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:43.097459 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-738845-6c477d4fb7-r4tnv" Oct 30 17:57:43.315576 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:43.315490 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-738845-867b76d77d-qzjng" Oct 30 17:57:43.497029 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:43.489609 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-738845-6c477d4fb7-r4tnv"] Oct 30 17:57:43.615114 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:43.615085 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-738845-6c477d4fb7-r4tnv" event={"ID":"dcb3f0ba-021a-4788-ad64-fcc9054fe5a1","Type":"ContainerStarted","Data":"912c16e5d693d332db7d70a406a744ca06e136dbe784f0a8fedd79d124d170e9"} Oct 30 17:57:43.701428 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:43.701394 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-738845-867b76d77d-qzjng"] Oct 30 17:57:43.701753 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:57:43.701725 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8abc8140_8a36_4cc7_80da_43738e049fe0.slice/crio-ce370e1c1426bbeaf85c3273630fe4c0307a90fa6f2f8ac906e3c5501248524a WatchSource:0}: Error finding container ce370e1c1426bbeaf85c3273630fe4c0307a90fa6f2f8ac906e3c5501248524a: Status 404 returned error can't find the container with id ce370e1c1426bbeaf85c3273630fe4c0307a90fa6f2f8ac906e3c5501248524a Oct 30 17:57:43.908259 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:43.908175 4965 patch_prober.go:29] interesting pod/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 17:57:43.908259 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:43.908227 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" podUID="6e3c3d72-4ad1-4368-93e4-5f429a7f80e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 17:57:44.625454 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:44.625414 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-738845-867b76d77d-qzjng" event={"ID":"8abc8140-8a36-4cc7-80da-43738e049fe0","Type":"ContainerStarted","Data":"c5ab5ce4cb9fe7248cb0a7c171ed70c94540182e0a81bd61540f805ddcff88c2"} Oct 30 17:57:44.625960 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:44.625475 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-738845-867b76d77d-qzjng" event={"ID":"8abc8140-8a36-4cc7-80da-43738e049fe0","Type":"ContainerStarted","Data":"ce370e1c1426bbeaf85c3273630fe4c0307a90fa6f2f8ac906e3c5501248524a"} Oct 30 17:57:44.625960 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:44.625500 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-738845-867b76d77d-qzjng" Oct 30 17:57:44.627159 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:44.627134 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-738845-6c477d4fb7-r4tnv" event={"ID":"dcb3f0ba-021a-4788-ad64-fcc9054fe5a1","Type":"ContainerStarted","Data":"2055b4612290f8e5685b3f98cdaf2f5bdb8b5801daedccae1ee5434ef1633b03"} Oct 30 17:57:44.627282 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:44.627268 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-738845-6c477d4fb7-r4tnv" Oct 30 17:57:44.669572 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:44.669524 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-738845-867b76d77d-qzjng" podStartSLOduration=2.669510142 podStartE2EDuration="2.669510142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:44.66771686 +0000 UTC m=+2722.719817008" watchObservedRunningTime="2026-10-30 17:57:44.669510142 +0000 UTC m=+2722.721610072" Oct 30 17:57:44.707895 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:44.707830 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-738845-6c477d4fb7-r4tnv" podStartSLOduration=2.7078172719999998 podStartE2EDuration="2.707817272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:44.706959042 +0000 UTC m=+2722.759059000" watchObservedRunningTime="2026-10-30 17:57:44.707817272 +0000 UTC m=+2722.759917203" Oct 30 17:57:44.912794 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:44.912716 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:44.913245 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:44.913230 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:44.913978 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:44.913960 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 17:57:46.732370 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:46.732337 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-455e4a-58cb69f468-dgb8f"] Oct 30 17:57:46.737416 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:46.737396 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-58cb69f468-dgb8f" Oct 30 17:57:46.757301 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:46.757270 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-455e4a-58cb69f468-dgb8f"] Oct 30 17:57:46.763609 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:46.763571 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-455e4a\"" Oct 30 17:57:46.795732 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:46.795689 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64608598-ae4c-4c7a-b1d7-6cc61414074e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-455e4a-58cb69f468-dgb8f\" (UID: \"64608598-ae4c-4c7a-b1d7-6cc61414074e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-58cb69f468-dgb8f" Oct 30 17:57:46.795945 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:46.795914 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq4dx\" (UniqueName: \"kubernetes.io/projected/64608598-ae4c-4c7a-b1d7-6cc61414074e-kube-api-access-cq4dx\") pod \"payload-processing-e2e-ipp-a-455e4a-58cb69f468-dgb8f\" (UID: \"64608598-ae4c-4c7a-b1d7-6cc61414074e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-58cb69f468-dgb8f" Oct 30 17:57:46.908889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:46.908006 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq4dx\" (UniqueName: \"kubernetes.io/projected/64608598-ae4c-4c7a-b1d7-6cc61414074e-kube-api-access-cq4dx\") pod \"payload-processing-e2e-ipp-a-455e4a-58cb69f468-dgb8f\" (UID: \"64608598-ae4c-4c7a-b1d7-6cc61414074e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-58cb69f468-dgb8f" Oct 30 17:57:46.908889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:46.908355 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64608598-ae4c-4c7a-b1d7-6cc61414074e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-455e4a-58cb69f468-dgb8f\" (UID: \"64608598-ae4c-4c7a-b1d7-6cc61414074e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-58cb69f468-dgb8f" Oct 30 17:57:46.910302 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:46.910212 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64608598-ae4c-4c7a-b1d7-6cc61414074e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-455e4a-58cb69f468-dgb8f\" (UID: \"64608598-ae4c-4c7a-b1d7-6cc61414074e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-58cb69f468-dgb8f" Oct 30 17:57:46.938138 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:46.938075 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq4dx\" (UniqueName: \"kubernetes.io/projected/64608598-ae4c-4c7a-b1d7-6cc61414074e-kube-api-access-cq4dx\") pod \"payload-processing-e2e-ipp-a-455e4a-58cb69f468-dgb8f\" (UID: \"64608598-ae4c-4c7a-b1d7-6cc61414074e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-58cb69f468-dgb8f" Oct 30 17:57:47.061010 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:47.057550 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-58cb69f468-dgb8f" Oct 30 17:57:47.377336 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:47.377309 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-455e4a-58cb69f468-dgb8f"] Oct 30 17:57:47.378589 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:57:47.378563 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64608598_ae4c_4c7a_b1d7_6cc61414074e.slice/crio-8a9e752a7c9d0fded946a032ec001f74e7420e2211c51c43cd18343aeaccf475 WatchSource:0}: Error finding container 8a9e752a7c9d0fded946a032ec001f74e7420e2211c51c43cd18343aeaccf475: Status 404 returned error can't find the container with id 8a9e752a7c9d0fded946a032ec001f74e7420e2211c51c43cd18343aeaccf475 Oct 30 17:57:47.731944 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:47.731014 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-58cb69f468-dgb8f" event={"ID":"64608598-ae4c-4c7a-b1d7-6cc61414074e","Type":"ContainerStarted","Data":"0cf36c43130f14a7c4d27e15191f5f8e7b83afbb27c382d47b7b1ff6c08f49d7"} Oct 30 17:57:47.731944 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:47.731089 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-58cb69f468-dgb8f" event={"ID":"64608598-ae4c-4c7a-b1d7-6cc61414074e","Type":"ContainerStarted","Data":"8a9e752a7c9d0fded946a032ec001f74e7420e2211c51c43cd18343aeaccf475"} Oct 30 17:57:47.732680 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:47.732651 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-58cb69f468-dgb8f" Oct 30 17:57:47.774720 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:47.774659 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-58cb69f468-dgb8f" podStartSLOduration=1.774644339 podStartE2EDuration="1.774644339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:47.773163956 +0000 UTC m=+2725.825263903" watchObservedRunningTime="2026-10-30 17:57:47.774644339 +0000 UTC m=+2725.826744269" Oct 30 17:57:50.687050 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:50.687003 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-222.ec2.internal" podUID="67161c7401d4577fb0911752be0668c2" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Oct 30 17:57:55.640951 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:55.640919 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-738845-867b76d77d-qzjng" Oct 30 17:57:55.641425 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:55.641083 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-738845-6c477d4fb7-r4tnv" Oct 30 17:57:59.750331 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:57:59.750298 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-58cb69f468-dgb8f" Oct 30 17:58:06.008130 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:58:06.008026 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:58:10.838961 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:10.838924 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-21870f-66f8f9975b-56cqf"] Oct 30 17:58:10.870059 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:10.870027 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-65cf8b4b8c-d4nqw"] Oct 30 17:58:12.268056 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:12.268022 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fab25f-f4c544ccb-j2kng"] Oct 30 17:58:12.274125 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:12.274103 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-f4c544ccb-j2kng" Oct 30 17:58:12.291261 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:12.291229 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fab25f-f4c544ccb-j2kng"] Oct 30 17:58:12.297626 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:12.297598 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-fab25f\"" Oct 30 17:58:12.312365 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:12.312256 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-8498f7756c-zhxg6"] Oct 30 17:58:12.318382 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:12.318357 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-8498f7756c-zhxg6" Oct 30 17:58:12.321120 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:12.321088 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-fab25f-dockercfg-ckt7w\"" Oct 30 17:58:12.338206 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:12.334642 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-8498f7756c-zhxg6"] Oct 30 17:58:12.364220 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:12.364178 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9576164-357d-4a63-bedf-f01b139de233-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fab25f-f4c544ccb-j2kng\" (UID: \"f9576164-357d-4a63-bedf-f01b139de233\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-f4c544ccb-j2kng" Oct 30 17:58:12.364408 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:12.364293 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9dwc\" (UniqueName: \"kubernetes.io/projected/f9576164-357d-4a63-bedf-f01b139de233-kube-api-access-b9dwc\") pod \"payload-processing-e2e-ipp-a-fab25f-f4c544ccb-j2kng\" (UID: \"f9576164-357d-4a63-bedf-f01b139de233\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-f4c544ccb-j2kng" Oct 30 17:58:12.474259 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:12.474179 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9dwc\" (UniqueName: \"kubernetes.io/projected/f9576164-357d-4a63-bedf-f01b139de233-kube-api-access-b9dwc\") pod \"payload-processing-e2e-ipp-a-fab25f-f4c544ccb-j2kng\" (UID: \"f9576164-357d-4a63-bedf-f01b139de233\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-f4c544ccb-j2kng" Oct 30 17:58:12.485556 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:12.485522 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db82c\" (UniqueName: \"kubernetes.io/projected/59a5a8c6-a647-428d-b181-a0528017fa79-kube-api-access-db82c\") pod \"payload-pre-processing-e2e-ipp-a-fab25f-8498f7756c-zhxg6\" (UID: \"59a5a8c6-a647-428d-b181-a0528017fa79\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-8498f7756c-zhxg6" Oct 30 17:58:12.486067 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:12.486042 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59a5a8c6-a647-428d-b181-a0528017fa79-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fab25f-8498f7756c-zhxg6\" (UID: \"59a5a8c6-a647-428d-b181-a0528017fa79\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-8498f7756c-zhxg6" Oct 30 17:58:12.497154 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:12.497121 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9576164-357d-4a63-bedf-f01b139de233-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fab25f-f4c544ccb-j2kng\" (UID: \"f9576164-357d-4a63-bedf-f01b139de233\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-f4c544ccb-j2kng" Oct 30 17:58:12.523984 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:12.498231 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9576164-357d-4a63-bedf-f01b139de233-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fab25f-f4c544ccb-j2kng\" (UID: \"f9576164-357d-4a63-bedf-f01b139de233\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-f4c544ccb-j2kng" Oct 30 17:58:12.582323 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:12.580112 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9dwc\" (UniqueName: \"kubernetes.io/projected/f9576164-357d-4a63-bedf-f01b139de233-kube-api-access-b9dwc\") pod \"payload-processing-e2e-ipp-a-fab25f-f4c544ccb-j2kng\" (UID: \"f9576164-357d-4a63-bedf-f01b139de233\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-f4c544ccb-j2kng" Oct 30 17:58:12.591826 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:12.591792 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-f4c544ccb-j2kng" Oct 30 17:58:12.625707 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:12.625676 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db82c\" (UniqueName: \"kubernetes.io/projected/59a5a8c6-a647-428d-b181-a0528017fa79-kube-api-access-db82c\") pod \"payload-pre-processing-e2e-ipp-a-fab25f-8498f7756c-zhxg6\" (UID: \"59a5a8c6-a647-428d-b181-a0528017fa79\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-8498f7756c-zhxg6" Oct 30 17:58:12.625856 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:12.625776 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59a5a8c6-a647-428d-b181-a0528017fa79-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fab25f-8498f7756c-zhxg6\" (UID: \"59a5a8c6-a647-428d-b181-a0528017fa79\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-8498f7756c-zhxg6" Oct 30 17:58:12.626649 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:12.626625 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59a5a8c6-a647-428d-b181-a0528017fa79-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fab25f-8498f7756c-zhxg6\" (UID: \"59a5a8c6-a647-428d-b181-a0528017fa79\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-8498f7756c-zhxg6" Oct 30 17:58:12.648788 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:12.648756 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db82c\" (UniqueName: \"kubernetes.io/projected/59a5a8c6-a647-428d-b181-a0528017fa79-kube-api-access-db82c\") pod \"payload-pre-processing-e2e-ipp-a-fab25f-8498f7756c-zhxg6\" (UID: \"59a5a8c6-a647-428d-b181-a0528017fa79\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-8498f7756c-zhxg6" Oct 30 17:58:12.906365 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:58:12.906336 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9576164_357d_4a63_bedf_f01b139de233.slice/crio-ef92c74c9de29a594d6991faf5d73bc9fb886b73213eb2c15936bb5611c4e950 WatchSource:0}: Error finding container ef92c74c9de29a594d6991faf5d73bc9fb886b73213eb2c15936bb5611c4e950: Status 404 returned error can't find the container with id ef92c74c9de29a594d6991faf5d73bc9fb886b73213eb2c15936bb5611c4e950 Oct 30 17:58:12.906627 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:12.906602 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fab25f-f4c544ccb-j2kng"] Oct 30 17:58:12.941058 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:12.941035 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-8498f7756c-zhxg6" Oct 30 17:58:12.994297 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:12.994265 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-f4c544ccb-j2kng" event={"ID":"f9576164-357d-4a63-bedf-f01b139de233","Type":"ContainerStarted","Data":"ef92c74c9de29a594d6991faf5d73bc9fb886b73213eb2c15936bb5611c4e950"} Oct 30 17:58:13.253778 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:58:13.253752 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59a5a8c6_a647_428d_b181_a0528017fa79.slice/crio-e624655052085d92177f06f4ed98f1bb4b72a9dcb17787e5a2ade956c23ca330 WatchSource:0}: Error finding container e624655052085d92177f06f4ed98f1bb4b72a9dcb17787e5a2ade956c23ca330: Status 404 returned error can't find the container with id e624655052085d92177f06f4ed98f1bb4b72a9dcb17787e5a2ade956c23ca330 Oct 30 17:58:13.253916 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:13.253775 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-8498f7756c-zhxg6"] Oct 30 17:58:14.005082 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:14.005045 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-f4c544ccb-j2kng" event={"ID":"f9576164-357d-4a63-bedf-f01b139de233","Type":"ContainerStarted","Data":"9cd3684c7418641b8b3096bae6bdcdd7d7a543d059d7834c72340ee6e2645666"} Oct 30 17:58:14.005590 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:14.005109 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-f4c544ccb-j2kng" Oct 30 17:58:14.006843 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:14.006807 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-8498f7756c-zhxg6" event={"ID":"59a5a8c6-a647-428d-b181-a0528017fa79","Type":"ContainerStarted","Data":"2ef70466c9ce37dbcfb81a680801db71da93e4deb67b2f092b10b9b9713d66b2"} Oct 30 17:58:14.007018 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:14.006877 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-8498f7756c-zhxg6" Oct 30 17:58:14.007018 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:14.006902 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-8498f7756c-zhxg6" event={"ID":"59a5a8c6-a647-428d-b181-a0528017fa79","Type":"ContainerStarted","Data":"e624655052085d92177f06f4ed98f1bb4b72a9dcb17787e5a2ade956c23ca330"} Oct 30 17:58:14.073019 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:14.062403 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-f4c544ccb-j2kng" podStartSLOduration=2.062376357 podStartE2EDuration="2.062376357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:14.047318807 +0000 UTC m=+2752.099418759" watchObservedRunningTime="2026-10-30 17:58:14.062376357 +0000 UTC m=+2752.114476290" Oct 30 17:58:15.839493 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:15.839452 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-66f8f9975b-56cqf" podUID="a8a3ced7-cbc1-457b-8004-a0175518e383" containerName="payload-processing" containerID="cri-o://a7aad68be56286b60413a401585f2f2976b6fa7b5bad25f2a4bfbfbcfc95b038" gracePeriod=25 Oct 30 17:58:15.871061 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:15.871012 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-65cf8b4b8c-d4nqw" podUID="28fdef67-ee11-47ab-a45a-d7c62873fbea" containerName="payload-pre-processing" containerID="cri-o://b4806c51972ac28f8b69f107a9d68beb59398caf0c967700955a82c476db377e" gracePeriod=25 Oct 30 17:58:16.025113 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:16.025076 4965 generic.go:356] "Generic (PLEG): container finished" podID="a8a3ced7-cbc1-457b-8004-a0175518e383" containerID="a7aad68be56286b60413a401585f2f2976b6fa7b5bad25f2a4bfbfbcfc95b038" exitCode=0 Oct 30 17:58:16.025340 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:16.025320 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-66f8f9975b-56cqf" event={"ID":"a8a3ced7-cbc1-457b-8004-a0175518e383","Type":"ContainerDied","Data":"a7aad68be56286b60413a401585f2f2976b6fa7b5bad25f2a4bfbfbcfc95b038"} Oct 30 17:58:16.027252 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:16.027226 4965 generic.go:356] "Generic (PLEG): container finished" podID="28fdef67-ee11-47ab-a45a-d7c62873fbea" containerID="b4806c51972ac28f8b69f107a9d68beb59398caf0c967700955a82c476db377e" exitCode=0 Oct 30 17:58:16.027423 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:16.027360 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-65cf8b4b8c-d4nqw" event={"ID":"28fdef67-ee11-47ab-a45a-d7c62873fbea","Type":"ContainerDied","Data":"b4806c51972ac28f8b69f107a9d68beb59398caf0c967700955a82c476db377e"} Oct 30 17:58:16.202468 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:16.202440 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-66f8f9975b-56cqf" Oct 30 17:58:16.206235 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:16.206214 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-65cf8b4b8c-d4nqw" Oct 30 17:58:16.212937 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:16.212914 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8a3ced7-cbc1-457b-8004-a0175518e383-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8a3ced7-cbc1-457b-8004-a0175518e383-plugins-config-volume\") pod \"a8a3ced7-cbc1-457b-8004-a0175518e383\" (UID: \"a8a3ced7-cbc1-457b-8004-a0175518e383\") " Oct 30 17:58:16.213072 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:16.213056 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28fdef67-ee11-47ab-a45a-d7c62873fbea-kube-api-access-hr8m7\" (UniqueName: \"kubernetes.io/projected/28fdef67-ee11-47ab-a45a-d7c62873fbea-kube-api-access-hr8m7\") pod \"28fdef67-ee11-47ab-a45a-d7c62873fbea\" (UID: \"28fdef67-ee11-47ab-a45a-d7c62873fbea\") " Oct 30 17:58:16.213139 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:16.213100 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28fdef67-ee11-47ab-a45a-d7c62873fbea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28fdef67-ee11-47ab-a45a-d7c62873fbea-plugins-config-volume\") pod \"28fdef67-ee11-47ab-a45a-d7c62873fbea\" (UID: \"28fdef67-ee11-47ab-a45a-d7c62873fbea\") " Oct 30 17:58:16.213240 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:16.213227 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8a3ced7-cbc1-457b-8004-a0175518e383-kube-api-access-vgdbb\" (UniqueName: \"kubernetes.io/projected/a8a3ced7-cbc1-457b-8004-a0175518e383-kube-api-access-vgdbb\") pod \"a8a3ced7-cbc1-457b-8004-a0175518e383\" (UID: \"a8a3ced7-cbc1-457b-8004-a0175518e383\") " Oct 30 17:58:16.213307 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:16.213243 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8a3ced7-cbc1-457b-8004-a0175518e383-plugins-config-volume" pod "a8a3ced7-cbc1-457b-8004-a0175518e383" (UID: "a8a3ced7-cbc1-457b-8004-a0175518e383"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:58:16.213532 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:16.213508 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28fdef67-ee11-47ab-a45a-d7c62873fbea-plugins-config-volume" pod "28fdef67-ee11-47ab-a45a-d7c62873fbea" (UID: "28fdef67-ee11-47ab-a45a-d7c62873fbea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:58:16.213753 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:16.213736 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28fdef67-ee11-47ab-a45a-d7c62873fbea-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:58:16.213809 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:16.213759 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8a3ced7-cbc1-457b-8004-a0175518e383-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:58:16.215204 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:16.215178 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28fdef67-ee11-47ab-a45a-d7c62873fbea-kube-api-access-hr8m7" pod "28fdef67-ee11-47ab-a45a-d7c62873fbea" (UID: "28fdef67-ee11-47ab-a45a-d7c62873fbea"). InnerVolumeSpecName "kube-api-access-hr8m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:58:16.215299 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:16.215236 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a3ced7-cbc1-457b-8004-a0175518e383-kube-api-access-vgdbb" pod "a8a3ced7-cbc1-457b-8004-a0175518e383" (UID: "a8a3ced7-cbc1-457b-8004-a0175518e383"). InnerVolumeSpecName "kube-api-access-vgdbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:58:16.246325 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:16.246280 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-8498f7756c-zhxg6" podStartSLOduration=4.246266709 podStartE2EDuration="4.246266709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:14.093661111 +0000 UTC m=+2752.145761041" watchObservedRunningTime="2026-10-30 17:58:16.246266709 +0000 UTC m=+2754.298366682" Oct 30 17:58:16.314707 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:16.314670 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr8m7\" (UniqueName: \"kubernetes.io/projected/28fdef67-ee11-47ab-a45a-d7c62873fbea-kube-api-access-hr8m7\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:58:16.314707 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:16.314700 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgdbb\" (UniqueName: \"kubernetes.io/projected/a8a3ced7-cbc1-457b-8004-a0175518e383-kube-api-access-vgdbb\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:58:17.036653 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:17.036620 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-65cf8b4b8c-d4nqw" Oct 30 17:58:17.037159 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:17.036656 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-65cf8b4b8c-d4nqw" event={"ID":"28fdef67-ee11-47ab-a45a-d7c62873fbea","Type":"ContainerDied","Data":"88018755012b94b92c35f8731bc23e7bb5161fa6ba8710074827f5cc29d686d9"} Oct 30 17:58:17.037159 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:17.036716 4965 scope.go:122] "RemoveContainer" containerID="b4806c51972ac28f8b69f107a9d68beb59398caf0c967700955a82c476db377e" Oct 30 17:58:17.038361 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:17.038341 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-66f8f9975b-56cqf" Oct 30 17:58:17.038441 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:17.038385 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-21870f-66f8f9975b-56cqf" event={"ID":"a8a3ced7-cbc1-457b-8004-a0175518e383","Type":"ContainerDied","Data":"dbb32172f987b31b0c23edbd6341e115da3c60a86b4ae33042c24d9297c7434b"} Oct 30 17:58:17.048768 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:17.048744 4965 scope.go:122] "RemoveContainer" containerID="a7aad68be56286b60413a401585f2f2976b6fa7b5bad25f2a4bfbfbcfc95b038" Oct 30 17:58:17.102580 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:17.102547 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-21870f-66f8f9975b-56cqf"] Oct 30 17:58:17.114742 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:17.114676 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-21870f-66f8f9975b-56cqf"] Oct 30 17:58:17.142087 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:17.142057 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-65cf8b4b8c-d4nqw"] Oct 30 17:58:17.146128 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:17.146107 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-65cf8b4b8c-d4nqw"] Oct 30 17:58:18.100671 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:18.100639 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28fdef67-ee11-47ab-a45a-d7c62873fbea" path="/var/lib/kubelet/pods/28fdef67-ee11-47ab-a45a-d7c62873fbea/volumes" Oct 30 17:58:18.101089 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:18.101074 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8a3ced7-cbc1-457b-8004-a0175518e383" path="/var/lib/kubelet/pods/a8a3ced7-cbc1-457b-8004-a0175518e383/volumes" Oct 30 17:58:25.039889 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:25.032030 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-8498f7756c-zhxg6" Oct 30 17:58:25.048891 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:25.044328 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-f4c544ccb-j2kng" Oct 30 17:58:25.105894 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:25.100916 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:58:36.018006 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:58:36.017970 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:58:50.445697 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:50.445655 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-bc6c5656c-mp82n"] Oct 30 17:58:50.447497 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:50.447468 4965 cpu_manager.go:414] "containerMap: removing container" podUID="28fdef67-ee11-47ab-a45a-d7c62873fbea" containerName="payload-pre-processing" Oct 30 17:58:50.447654 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:50.447642 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28fdef67-ee11-47ab-a45a-d7c62873fbea" containerName="payload-pre-processing" Oct 30 17:58:50.447746 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:50.447737 4965 cpu_manager.go:414] "containerMap: removing container" podUID="a8a3ced7-cbc1-457b-8004-a0175518e383" containerName="payload-processing" Oct 30 17:58:50.447817 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:50.447808 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8a3ced7-cbc1-457b-8004-a0175518e383" containerName="payload-processing" Oct 30 17:58:50.452236 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:50.452209 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="28fdef67-ee11-47ab-a45a-d7c62873fbea" containerName="payload-pre-processing" Oct 30 17:58:50.452411 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:50.452398 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8a3ced7-cbc1-457b-8004-a0175518e383" containerName="payload-processing" Oct 30 17:58:50.511586 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:50.511555 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-bc6c5656c-mp82n"] Oct 30 17:58:50.511917 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:50.511841 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-bc6c5656c-mp82n" Oct 30 17:58:50.541797 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:50.541760 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-21870f\"" Oct 30 17:58:50.541797 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:50.541767 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-21870f-dockercfg-qvphd\"" Oct 30 17:58:50.616453 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:50.616420 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvtgj\" (UniqueName: \"kubernetes.io/projected/fa4299c9-7f64-4e69-8f31-0499a36a4179-kube-api-access-nvtgj\") pod \"payload-pre-processing-e2e-ipp-a-21870f-bc6c5656c-mp82n\" (UID: \"fa4299c9-7f64-4e69-8f31-0499a36a4179\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-bc6c5656c-mp82n" Oct 30 17:58:50.616625 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:50.616553 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4299c9-7f64-4e69-8f31-0499a36a4179-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-21870f-bc6c5656c-mp82n\" (UID: \"fa4299c9-7f64-4e69-8f31-0499a36a4179\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-bc6c5656c-mp82n" Oct 30 17:58:50.718450 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:50.718369 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvtgj\" (UniqueName: \"kubernetes.io/projected/fa4299c9-7f64-4e69-8f31-0499a36a4179-kube-api-access-nvtgj\") pod \"payload-pre-processing-e2e-ipp-a-21870f-bc6c5656c-mp82n\" (UID: \"fa4299c9-7f64-4e69-8f31-0499a36a4179\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-bc6c5656c-mp82n" Oct 30 17:58:50.718601 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:50.718462 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4299c9-7f64-4e69-8f31-0499a36a4179-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-21870f-bc6c5656c-mp82n\" (UID: \"fa4299c9-7f64-4e69-8f31-0499a36a4179\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-bc6c5656c-mp82n" Oct 30 17:58:50.719257 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:50.719236 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4299c9-7f64-4e69-8f31-0499a36a4179-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-21870f-bc6c5656c-mp82n\" (UID: \"fa4299c9-7f64-4e69-8f31-0499a36a4179\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-bc6c5656c-mp82n" Oct 30 17:58:50.741628 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:50.741602 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvtgj\" (UniqueName: \"kubernetes.io/projected/fa4299c9-7f64-4e69-8f31-0499a36a4179-kube-api-access-nvtgj\") pod \"payload-pre-processing-e2e-ipp-a-21870f-bc6c5656c-mp82n\" (UID: \"fa4299c9-7f64-4e69-8f31-0499a36a4179\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-bc6c5656c-mp82n" Oct 30 17:58:50.838982 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:50.838949 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-bc6c5656c-mp82n" Oct 30 17:58:51.191040 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:58:51.190979 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa4299c9_7f64_4e69_8f31_0499a36a4179.slice/crio-a538cd19513c18a7c9e57bd55c5e65ec6779ca751190c89c7d4d2bf29216efb2 WatchSource:0}: Error finding container a538cd19513c18a7c9e57bd55c5e65ec6779ca751190c89c7d4d2bf29216efb2: Status 404 returned error can't find the container with id a538cd19513c18a7c9e57bd55c5e65ec6779ca751190c89c7d4d2bf29216efb2 Oct 30 17:58:51.193403 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:51.193384 4965 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:58:51.195317 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:51.195292 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-bc6c5656c-mp82n"] Oct 30 17:58:51.454359 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:51.454278 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-bc6c5656c-mp82n" event={"ID":"fa4299c9-7f64-4e69-8f31-0499a36a4179","Type":"ContainerStarted","Data":"a538cd19513c18a7c9e57bd55c5e65ec6779ca751190c89c7d4d2bf29216efb2"} Oct 30 17:58:53.472443 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:53.472397 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-bc6c5656c-mp82n" event={"ID":"fa4299c9-7f64-4e69-8f31-0499a36a4179","Type":"ContainerStarted","Data":"e7686aef5d7aa5530a25d6817eae28a89abba77a697befece07553ea67abcc4d"} Oct 30 17:58:53.472859 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:53.472480 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-bc6c5656c-mp82n" Oct 30 17:58:53.516107 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:58:53.516051 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-bc6c5656c-mp82n" podStartSLOduration=1.76520382 podStartE2EDuration="3.516036432s" totalImagesPullingTime="1.750832612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:53.512067761 +0000 UTC m=+2791.564167692" watchObservedRunningTime="2026-10-30 17:58:53.516036432 +0000 UTC m=+2791.568136363" Oct 30 17:59:04.487517 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:04.487484 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-bc6c5656c-mp82n" Oct 30 17:59:05.997663 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:59:05.997623 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:59:16.787947 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:16.787909 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-455e4a-58cb69f468-dgb8f"] Oct 30 17:59:21.789234 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:21.789177 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-58cb69f468-dgb8f" podUID="64608598-ae4c-4c7a-b1d7-6cc61414074e" containerName="payload-processing" containerID="cri-o://0cf36c43130f14a7c4d27e15191f5f8e7b83afbb27c382d47b7b1ff6c08f49d7" gracePeriod=25 Oct 30 17:59:22.060174 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.059376 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj"] Oct 30 17:59:22.078318 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.078291 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj"] Oct 30 17:59:22.078605 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.078592 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:22.094232 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.094207 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-738845-openshift-default-dockercfg-5lqwd\"" Oct 30 17:59:22.171034 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.171007 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-58cb69f468-dgb8f" Oct 30 17:59:22.181552 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.181523 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64608598-ae4c-4c7a-b1d7-6cc61414074e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64608598-ae4c-4c7a-b1d7-6cc61414074e-plugins-config-volume\") pod \"64608598-ae4c-4c7a-b1d7-6cc61414074e\" (UID: \"64608598-ae4c-4c7a-b1d7-6cc61414074e\") " Oct 30 17:59:22.181698 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.181568 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64608598-ae4c-4c7a-b1d7-6cc61414074e-kube-api-access-cq4dx\" (UniqueName: \"kubernetes.io/projected/64608598-ae4c-4c7a-b1d7-6cc61414074e-kube-api-access-cq4dx\") pod \"64608598-ae4c-4c7a-b1d7-6cc61414074e\" (UID: \"64608598-ae4c-4c7a-b1d7-6cc61414074e\") " Oct 30 17:59:22.181888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.181850 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64608598-ae4c-4c7a-b1d7-6cc61414074e-plugins-config-volume" pod "64608598-ae4c-4c7a-b1d7-6cc61414074e" (UID: "64608598-ae4c-4c7a-b1d7-6cc61414074e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:59:22.181944 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.181924 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-workload-socket\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:22.182133 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.182111 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istio-envoy\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:22.182223 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.182157 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istio-token\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:22.182294 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.182262 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-workload-certs\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:22.182360 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.182301 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istio-data\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:22.182360 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.182318 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl99r\" (UniqueName: \"kubernetes.io/projected/d43a70eb-9850-4990-8385-fd2e2aab5bc2-kube-api-access-jl99r\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:22.182582 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.182558 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istio-podinfo\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:22.182661 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.182615 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istiod-ca-cert\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:22.182661 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.182645 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-credential-socket\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:22.182996 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.182975 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64608598-ae4c-4c7a-b1d7-6cc61414074e-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:59:22.184087 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.184063 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64608598-ae4c-4c7a-b1d7-6cc61414074e-kube-api-access-cq4dx" pod "64608598-ae4c-4c7a-b1d7-6cc61414074e" (UID: "64608598-ae4c-4c7a-b1d7-6cc61414074e"). InnerVolumeSpecName "kube-api-access-cq4dx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:59:22.284417 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.284383 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istio-envoy\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:22.284625 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.284432 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istio-token\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:22.284625 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.284535 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-workload-certs\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:22.284625 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.284562 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istio-data\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:22.284625 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.284579 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl99r\" (UniqueName: \"kubernetes.io/projected/d43a70eb-9850-4990-8385-fd2e2aab5bc2-kube-api-access-jl99r\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:22.284879 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.284659 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istio-podinfo\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:22.284879 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.284689 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istiod-ca-cert\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:22.284879 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.284708 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-credential-socket\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:22.284879 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.284840 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-workload-socket\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:22.285108 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.284985 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq4dx\" (UniqueName: \"kubernetes.io/projected/64608598-ae4c-4c7a-b1d7-6cc61414074e-kube-api-access-cq4dx\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:59:22.285108 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.284983 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istio-data\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:22.285108 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.285021 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-workload-certs\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:22.285277 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.285188 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-credential-socket\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:22.285277 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.285213 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-workload-socket\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:22.285454 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.285430 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istiod-ca-cert\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:22.286878 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.286834 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istio-envoy\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:22.287163 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.287145 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istio-podinfo\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:22.307838 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.307814 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istio-token\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:22.308011 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.307992 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl99r\" (UniqueName: \"kubernetes.io/projected/d43a70eb-9850-4990-8385-fd2e2aab5bc2-kube-api-access-jl99r\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:22.414841 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.414810 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-738845-openshift-default-dockercfg-5lqwd\"" Oct 30 17:59:22.421327 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.421302 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:22.793599 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:59:22.793565 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd43a70eb_9850_4990_8385_fd2e2aab5bc2.slice/crio-50f10972378734137f7eeccac67125920083adc8a654e170e7a3540fc646b79b WatchSource:0}: Error finding container 50f10972378734137f7eeccac67125920083adc8a654e170e7a3540fc646b79b: Status 404 returned error can't find the container with id 50f10972378734137f7eeccac67125920083adc8a654e170e7a3540fc646b79b Oct 30 17:59:22.794218 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.794196 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj"] Oct 30 17:59:22.797550 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.797503 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:59:22.797654 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.797586 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:59:22.797654 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.797632 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:59:22.803266 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.803233 4965 generic.go:356] "Generic (PLEG): container finished" podID="64608598-ae4c-4c7a-b1d7-6cc61414074e" containerID="0cf36c43130f14a7c4d27e15191f5f8e7b83afbb27c382d47b7b1ff6c08f49d7" exitCode=0 Oct 30 17:59:22.803385 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.803274 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-58cb69f468-dgb8f" Oct 30 17:59:22.803385 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.803312 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-58cb69f468-dgb8f" event={"ID":"64608598-ae4c-4c7a-b1d7-6cc61414074e","Type":"ContainerDied","Data":"0cf36c43130f14a7c4d27e15191f5f8e7b83afbb27c382d47b7b1ff6c08f49d7"} Oct 30 17:59:22.803385 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.803371 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-58cb69f468-dgb8f" event={"ID":"64608598-ae4c-4c7a-b1d7-6cc61414074e","Type":"ContainerDied","Data":"8a9e752a7c9d0fded946a032ec001f74e7420e2211c51c43cd18343aeaccf475"} Oct 30 17:59:22.803496 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.803405 4965 scope.go:122] "RemoveContainer" containerID="0cf36c43130f14a7c4d27e15191f5f8e7b83afbb27c382d47b7b1ff6c08f49d7" Oct 30 17:59:22.836377 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.836360 4965 scope.go:122] "RemoveContainer" containerID="0cf36c43130f14a7c4d27e15191f5f8e7b83afbb27c382d47b7b1ff6c08f49d7" Oct 30 17:59:22.836671 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:59:22.836647 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cf36c43130f14a7c4d27e15191f5f8e7b83afbb27c382d47b7b1ff6c08f49d7\": container with ID starting with 0cf36c43130f14a7c4d27e15191f5f8e7b83afbb27c382d47b7b1ff6c08f49d7 not found: ID does not exist" containerID="0cf36c43130f14a7c4d27e15191f5f8e7b83afbb27c382d47b7b1ff6c08f49d7" Oct 30 17:59:22.836723 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.836679 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cf36c43130f14a7c4d27e15191f5f8e7b83afbb27c382d47b7b1ff6c08f49d7"} err="failed to get container status \"0cf36c43130f14a7c4d27e15191f5f8e7b83afbb27c382d47b7b1ff6c08f49d7\": rpc error: code = NotFound desc = could not find container \"0cf36c43130f14a7c4d27e15191f5f8e7b83afbb27c382d47b7b1ff6c08f49d7\": container with ID starting with 0cf36c43130f14a7c4d27e15191f5f8e7b83afbb27c382d47b7b1ff6c08f49d7 not found: ID does not exist" Oct 30 17:59:22.891659 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.891633 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-455e4a-58cb69f468-dgb8f"] Oct 30 17:59:22.897196 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:22.897166 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-455e4a-58cb69f468-dgb8f"] Oct 30 17:59:23.813749 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:23.813712 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" event={"ID":"d43a70eb-9850-4990-8385-fd2e2aab5bc2","Type":"ContainerStarted","Data":"429f4ddfe1d4a6f4a8b8688486d397f86e1536b5c3225f91d069980411be3b72"} Oct 30 17:59:23.814195 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:23.813812 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" event={"ID":"d43a70eb-9850-4990-8385-fd2e2aab5bc2","Type":"ContainerStarted","Data":"50f10972378734137f7eeccac67125920083adc8a654e170e7a3540fc646b79b"} Oct 30 17:59:23.876885 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:23.876761 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" podStartSLOduration=2.876742326 podStartE2EDuration="2.876742326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:59:23.875269691 +0000 UTC m=+2821.927369623" watchObservedRunningTime="2026-10-30 17:59:23.876742326 +0000 UTC m=+2821.928842258" Oct 30 17:59:24.100697 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:24.100666 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64608598-ae4c-4c7a-b1d7-6cc61414074e" path="/var/lib/kubelet/pods/64608598-ae4c-4c7a-b1d7-6cc61414074e/volumes" Oct 30 17:59:24.421998 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:24.421892 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:24.423446 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:24.423401 4965 patch_prober.go:29] interesting pod/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 17:59:24.423590 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:24.423457 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" podUID="d43a70eb-9850-4990-8385-fd2e2aab5bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 17:59:25.421617 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:25.421578 4965 patch_prober.go:29] interesting pod/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 17:59:25.422130 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:25.421633 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" podUID="d43a70eb-9850-4990-8385-fd2e2aab5bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 17:59:26.422930 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:26.422705 4965 patch_prober.go:29] interesting pod/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 17:59:26.422930 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:26.422765 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" podUID="d43a70eb-9850-4990-8385-fd2e2aab5bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 17:59:27.422078 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:27.422046 4965 patch_prober.go:29] interesting pod/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 17:59:27.422265 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:27.422103 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" podUID="d43a70eb-9850-4990-8385-fd2e2aab5bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 17:59:28.289399 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:28.289361 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk"] Oct 30 17:59:28.289916 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:28.289611 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" podUID="22ed5344-88c7-4530-bcbf-327df41c4492" containerName="istio-proxy" containerID="cri-o://e4b11dc2139671848fb8a7134b7ba9c9d287bc24862591f2c34cea0f32ea2cfc" gracePeriod=30 Oct 30 17:59:28.421823 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:28.421787 4965 patch_prober.go:29] interesting pod/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 17:59:28.422048 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:28.421851 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" podUID="d43a70eb-9850-4990-8385-fd2e2aab5bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 17:59:29.426263 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:29.426224 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:29.426723 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:29.426711 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:29.427390 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:29.427364 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 17:59:29.975794 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:29.975756 4965 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:59:29.975982 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:29.975814 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" podUID="22ed5344-88c7-4530-bcbf-327df41c4492" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:59:33.585473 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.585447 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:59:33.774262 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.774222 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-credential-socket\") pod \"22ed5344-88c7-4530-bcbf-327df41c4492\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " Oct 30 17:59:33.774446 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.774360 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-workload-socket\") pod \"22ed5344-88c7-4530-bcbf-327df41c4492\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " Oct 30 17:59:33.774446 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.774411 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-istio-data\") pod \"22ed5344-88c7-4530-bcbf-327df41c4492\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " Oct 30 17:59:33.774540 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.774451 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22ed5344-88c7-4530-bcbf-327df41c4492-kube-api-access-bb8td\" (UniqueName: \"kubernetes.io/projected/22ed5344-88c7-4530-bcbf-327df41c4492-kube-api-access-bb8td\") pod \"22ed5344-88c7-4530-bcbf-327df41c4492\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " Oct 30 17:59:33.774540 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.774493 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22ed5344-88c7-4530-bcbf-327df41c4492-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22ed5344-88c7-4530-bcbf-327df41c4492-istiod-ca-cert\") pod \"22ed5344-88c7-4530-bcbf-327df41c4492\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " Oct 30 17:59:33.774540 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.774523 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22ed5344-88c7-4530-bcbf-327df41c4492-istio-token\" (UniqueName: \"kubernetes.io/projected/22ed5344-88c7-4530-bcbf-327df41c4492-istio-token\") pod \"22ed5344-88c7-4530-bcbf-327df41c4492\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " Oct 30 17:59:33.774707 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.774554 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-credential-socket" pod "22ed5344-88c7-4530-bcbf-327df41c4492" (UID: "22ed5344-88c7-4530-bcbf-327df41c4492"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:59:33.774707 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.774607 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-workload-certs\") pod \"22ed5344-88c7-4530-bcbf-327df41c4492\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " Oct 30 17:59:33.774707 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.774646 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-workload-socket" pod "22ed5344-88c7-4530-bcbf-327df41c4492" (UID: "22ed5344-88c7-4530-bcbf-327df41c4492"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:59:33.774707 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.774669 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22ed5344-88c7-4530-bcbf-327df41c4492-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22ed5344-88c7-4530-bcbf-327df41c4492-istio-podinfo\") pod \"22ed5344-88c7-4530-bcbf-327df41c4492\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " Oct 30 17:59:33.774707 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.774706 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-istio-envoy\") pod \"22ed5344-88c7-4530-bcbf-327df41c4492\" (UID: \"22ed5344-88c7-4530-bcbf-327df41c4492\") " Oct 30 17:59:33.774995 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.774775 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-istio-data" pod "22ed5344-88c7-4530-bcbf-327df41c4492" (UID: "22ed5344-88c7-4530-bcbf-327df41c4492"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:59:33.774995 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.774898 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-workload-certs" pod "22ed5344-88c7-4530-bcbf-327df41c4492" (UID: "22ed5344-88c7-4530-bcbf-327df41c4492"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:59:33.774995 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.774907 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22ed5344-88c7-4530-bcbf-327df41c4492-istiod-ca-cert" pod "22ed5344-88c7-4530-bcbf-327df41c4492" (UID: "22ed5344-88c7-4530-bcbf-327df41c4492"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:59:33.775476 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.775450 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:59:33.775602 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.775481 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:59:33.775602 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.775499 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:59:33.775602 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.775515 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22ed5344-88c7-4530-bcbf-327df41c4492-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:59:33.775602 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.775527 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:59:33.776926 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.776903 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22ed5344-88c7-4530-bcbf-327df41c4492-istio-podinfo" pod "22ed5344-88c7-4530-bcbf-327df41c4492" (UID: "22ed5344-88c7-4530-bcbf-327df41c4492"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:59:33.777105 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.776937 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22ed5344-88c7-4530-bcbf-327df41c4492-kube-api-access-bb8td" pod "22ed5344-88c7-4530-bcbf-327df41c4492" (UID: "22ed5344-88c7-4530-bcbf-327df41c4492"). InnerVolumeSpecName "kube-api-access-bb8td". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:59:33.777105 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.776962 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22ed5344-88c7-4530-bcbf-327df41c4492-istio-token" pod "22ed5344-88c7-4530-bcbf-327df41c4492" (UID: "22ed5344-88c7-4530-bcbf-327df41c4492"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:59:33.777105 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.777022 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-istio-envoy" pod "22ed5344-88c7-4530-bcbf-327df41c4492" (UID: "22ed5344-88c7-4530-bcbf-327df41c4492"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:59:33.876612 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.876576 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb8td\" (UniqueName: \"kubernetes.io/projected/22ed5344-88c7-4530-bcbf-327df41c4492-kube-api-access-bb8td\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:59:33.876612 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.876609 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22ed5344-88c7-4530-bcbf-327df41c4492-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:59:33.876612 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.876619 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22ed5344-88c7-4530-bcbf-327df41c4492-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:59:33.876946 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.876628 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22ed5344-88c7-4530-bcbf-327df41c4492-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:59:33.890262 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.890220 4965 generic.go:356] "Generic (PLEG): container finished" podID="22ed5344-88c7-4530-bcbf-327df41c4492" containerID="e4b11dc2139671848fb8a7134b7ba9c9d287bc24862591f2c34cea0f32ea2cfc" exitCode=0 Oct 30 17:59:33.890420 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.890294 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" Oct 30 17:59:33.890420 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.890303 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" event={"ID":"22ed5344-88c7-4530-bcbf-327df41c4492","Type":"ContainerDied","Data":"e4b11dc2139671848fb8a7134b7ba9c9d287bc24862591f2c34cea0f32ea2cfc"} Oct 30 17:59:33.890420 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.890398 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk" event={"ID":"22ed5344-88c7-4530-bcbf-327df41c4492","Type":"ContainerDied","Data":"15ea0f01a19e4fe6adbf3b6c878d84c0859f704c5a7bcd276c3364e12bd7bb33"} Oct 30 17:59:33.890555 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.890423 4965 scope.go:122] "RemoveContainer" containerID="e4b11dc2139671848fb8a7134b7ba9c9d287bc24862591f2c34cea0f32ea2cfc" Oct 30 17:59:33.902551 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.902530 4965 scope.go:122] "RemoveContainer" containerID="e4b11dc2139671848fb8a7134b7ba9c9d287bc24862591f2c34cea0f32ea2cfc" Oct 30 17:59:33.902799 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:59:33.902780 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4b11dc2139671848fb8a7134b7ba9c9d287bc24862591f2c34cea0f32ea2cfc\": container with ID starting with e4b11dc2139671848fb8a7134b7ba9c9d287bc24862591f2c34cea0f32ea2cfc not found: ID does not exist" containerID="e4b11dc2139671848fb8a7134b7ba9c9d287bc24862591f2c34cea0f32ea2cfc" Oct 30 17:59:33.902849 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.902807 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4b11dc2139671848fb8a7134b7ba9c9d287bc24862591f2c34cea0f32ea2cfc"} err="failed to get container status \"e4b11dc2139671848fb8a7134b7ba9c9d287bc24862591f2c34cea0f32ea2cfc\": rpc error: code = NotFound desc = could not find container \"e4b11dc2139671848fb8a7134b7ba9c9d287bc24862591f2c34cea0f32ea2cfc\": container with ID starting with e4b11dc2139671848fb8a7134b7ba9c9d287bc24862591f2c34cea0f32ea2cfc not found: ID does not exist" Oct 30 17:59:33.944543 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.944516 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk"] Oct 30 17:59:33.957190 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:33.957163 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-mvxrk"] Oct 30 17:59:34.101349 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:34.101317 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22ed5344-88c7-4530-bcbf-327df41c4492" path="/var/lib/kubelet/pods/22ed5344-88c7-4530-bcbf-327df41c4492/volumes" Oct 30 17:59:35.998473 ip-10-0-139-222 kubenswrapper[4965]: E1030 17:59:35.998427 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 17:59:41.586887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:41.585945 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 17:59:46.703182 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:46.703150 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fab25f-f4c544ccb-j2kng"] Oct 30 17:59:46.734039 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:46.734009 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-8498f7756c-zhxg6"] Oct 30 17:59:47.800046 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:47.800007 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd"] Oct 30 17:59:47.866009 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:47.865977 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv"] Oct 30 17:59:47.866806 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:47.866785 4965 cpu_manager.go:414] "containerMap: removing container" podUID="22ed5344-88c7-4530-bcbf-327df41c4492" containerName="istio-proxy" Oct 30 17:59:47.866806 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:47.866804 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22ed5344-88c7-4530-bcbf-327df41c4492" containerName="istio-proxy" Oct 30 17:59:47.867014 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:47.866817 4965 cpu_manager.go:414] "containerMap: removing container" podUID="64608598-ae4c-4c7a-b1d7-6cc61414074e" containerName="payload-processing" Oct 30 17:59:47.867014 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:47.866822 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64608598-ae4c-4c7a-b1d7-6cc61414074e" containerName="payload-processing" Oct 30 17:59:47.867014 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:47.866958 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="22ed5344-88c7-4530-bcbf-327df41c4492" containerName="istio-proxy" Oct 30 17:59:47.867014 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:47.866969 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="64608598-ae4c-4c7a-b1d7-6cc61414074e" containerName="payload-processing" Oct 30 17:59:47.872810 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:47.872786 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:47.892111 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:47.892083 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv"] Oct 30 17:59:48.028886 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.017123 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-23c1e8-openshift-default-dockercfg-4mhhv\"" Oct 30 17:59:48.048367 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.036840 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-credential-socket\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:48.048675 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.048640 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-workload-certs\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:48.048923 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.048898 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istio-data\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:48.049091 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.049072 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4fp5\" (UniqueName: \"kubernetes.io/projected/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-kube-api-access-f4fp5\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:48.049320 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.049300 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-workload-socket\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:48.049461 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.049445 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istiod-ca-cert\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:48.049954 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.049931 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istio-token\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:48.050147 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.050094 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istio-podinfo\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:48.050421 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.050399 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istio-envoy\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:48.152025 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.151987 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istio-token\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:48.152025 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.152025 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istio-podinfo\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:48.152251 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.152058 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istio-envoy\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:48.152251 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.152186 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-credential-socket\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:48.152251 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.152212 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-workload-certs\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:48.152383 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.152261 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istio-data\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:48.152383 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.152287 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4fp5\" (UniqueName: \"kubernetes.io/projected/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-kube-api-access-f4fp5\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:48.152494 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.152392 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-workload-socket\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:48.152494 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.152433 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istiod-ca-cert\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:48.152638 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.152611 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-credential-socket\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:48.152883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.152700 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-workload-certs\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:48.152883 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.152744 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istio-data\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:48.153015 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.152844 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-workload-socket\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:48.153139 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.153120 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istiod-ca-cert\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:48.155071 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.155048 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istio-envoy\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:48.155422 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.155382 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istio-podinfo\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:48.175541 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.175511 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istio-token\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:48.175757 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.175725 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4fp5\" (UniqueName: \"kubernetes.io/projected/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-kube-api-access-f4fp5\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:48.190490 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.190464 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:48.518217 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.518190 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv"] Oct 30 17:59:48.518831 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:59:48.518805 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod722e3954_7fe4_47c2_90f5_5d3b4eb5ca5b.slice/crio-6ecff86a615cf52d5836870a5069680d53c42a92c27413a2af034ebd255f5247 WatchSource:0}: Error finding container 6ecff86a615cf52d5836870a5069680d53c42a92c27413a2af034ebd255f5247: Status 404 returned error can't find the container with id 6ecff86a615cf52d5836870a5069680d53c42a92c27413a2af034ebd255f5247 Oct 30 17:59:48.521667 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.521627 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:59:48.521759 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.521708 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:59:48.521835 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:48.521753 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:59:49.091454 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:49.091416 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" event={"ID":"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b","Type":"ContainerStarted","Data":"55919ca08474befb4c087b7bd0d75e85118eb591dd5755982eb42a27ea686318"} Oct 30 17:59:49.091938 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:49.091468 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" event={"ID":"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b","Type":"ContainerStarted","Data":"6ecff86a615cf52d5836870a5069680d53c42a92c27413a2af034ebd255f5247"} Oct 30 17:59:49.152678 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:49.152630 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" podStartSLOduration=2.152618384 podStartE2EDuration="2.152618384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:59:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:59:49.152075781 +0000 UTC m=+2847.204175756" watchObservedRunningTime="2026-10-30 17:59:49.152618384 +0000 UTC m=+2847.204718315" Oct 30 17:59:49.191479 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:49.191447 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:49.193008 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:49.192987 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 17:59:49.193076 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:49.193033 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" podUID="722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 17:59:50.191618 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:50.191577 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 17:59:50.192090 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:50.191638 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" podUID="722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 17:59:51.191447 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.191413 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 17:59:51.191640 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.191463 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" podUID="722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 17:59:51.389789 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.389753 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd"] Oct 30 17:59:51.395377 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.395351 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 17:59:51.407453 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.407414 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/241c1434-3e5c-4120-8eb5-507bf30a92ec-istio-podinfo\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 17:59:51.407617 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.407512 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-workload-socket\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 17:59:51.407617 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.407603 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-workload-certs\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 17:59:51.407758 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.407640 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/241c1434-3e5c-4120-8eb5-507bf30a92ec-istio-token\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 17:59:51.407758 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.407675 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-istio-data\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 17:59:51.407758 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.407702 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcgfl\" (UniqueName: \"kubernetes.io/projected/241c1434-3e5c-4120-8eb5-507bf30a92ec-kube-api-access-xcgfl\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 17:59:51.407990 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.407780 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/241c1434-3e5c-4120-8eb5-507bf30a92ec-istiod-ca-cert\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 17:59:51.407990 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.407904 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-credential-socket\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 17:59:51.408605 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.408578 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd"] Oct 30 17:59:51.410698 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.410673 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-istio-envoy\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 17:59:51.488423 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.488324 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-fab25f-openshift-default-dockercfg-2rfnw\"" Oct 30 17:59:51.512477 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.512445 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-workload-socket\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 17:59:51.512648 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.512548 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-workload-certs\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 17:59:51.512648 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.512578 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/241c1434-3e5c-4120-8eb5-507bf30a92ec-istio-token\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 17:59:51.512648 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.512600 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-istio-data\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 17:59:51.512789 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.512651 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcgfl\" (UniqueName: \"kubernetes.io/projected/241c1434-3e5c-4120-8eb5-507bf30a92ec-kube-api-access-xcgfl\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 17:59:51.512789 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.512750 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/241c1434-3e5c-4120-8eb5-507bf30a92ec-istiod-ca-cert\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 17:59:51.512923 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.512855 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-credential-socket\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 17:59:51.512995 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.512927 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-workload-socket\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 17:59:51.512995 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.512965 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-istio-data\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 17:59:51.513161 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.512995 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-istio-envoy\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 17:59:51.513161 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.513068 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/241c1434-3e5c-4120-8eb5-507bf30a92ec-istio-podinfo\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 17:59:51.513161 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.513088 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-workload-certs\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 17:59:51.513322 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.513237 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-credential-socket\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 17:59:51.513571 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.513549 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/241c1434-3e5c-4120-8eb5-507bf30a92ec-istiod-ca-cert\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 17:59:51.515262 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.515235 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-istio-envoy\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 17:59:51.515475 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.515459 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/241c1434-3e5c-4120-8eb5-507bf30a92ec-istio-podinfo\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 17:59:51.536080 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.536051 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/241c1434-3e5c-4120-8eb5-507bf30a92ec-istio-token\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 17:59:51.536182 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.536137 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcgfl\" (UniqueName: \"kubernetes.io/projected/241c1434-3e5c-4120-8eb5-507bf30a92ec-kube-api-access-xcgfl\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 17:59:51.704021 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.703982 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-f4c544ccb-j2kng" podUID="f9576164-357d-4a63-bedf-f01b139de233" containerName="payload-processing" containerID="cri-o://9cd3684c7418641b8b3096bae6bdcdd7d7a543d059d7834c72340ee6e2645666" gracePeriod=25 Oct 30 17:59:51.722778 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.722744 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 17:59:51.734924 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:51.734849 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-8498f7756c-zhxg6" podUID="59a5a8c6-a647-428d-b181-a0528017fa79" containerName="payload-pre-processing" containerID="cri-o://2ef70466c9ce37dbcfb81a680801db71da93e4deb67b2f092b10b9b9713d66b2" gracePeriod=25 Oct 30 17:59:52.119812 ip-10-0-139-222 kubenswrapper[4965]: W1030 17:59:52.119780 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod241c1434_3e5c_4120_8eb5_507bf30a92ec.slice/crio-ba544a613cdf99bce0e87e927ea0342e6e75f3d92688222727723fdf9c776ff1 WatchSource:0}: Error finding container ba544a613cdf99bce0e87e927ea0342e6e75f3d92688222727723fdf9c776ff1: Status 404 returned error can't find the container with id ba544a613cdf99bce0e87e927ea0342e6e75f3d92688222727723fdf9c776ff1 Oct 30 17:59:52.129024 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:52.128976 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:59:52.129204 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:52.129077 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:59:52.129204 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:52.129123 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 17:59:52.132072 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:52.132050 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd"] Oct 30 17:59:52.140695 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:52.140544 4965 generic.go:356] "Generic (PLEG): container finished" podID="f9576164-357d-4a63-bedf-f01b139de233" containerID="9cd3684c7418641b8b3096bae6bdcdd7d7a543d059d7834c72340ee6e2645666" exitCode=0 Oct 30 17:59:52.140695 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:52.140652 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-f4c544ccb-j2kng" event={"ID":"f9576164-357d-4a63-bedf-f01b139de233","Type":"ContainerDied","Data":"9cd3684c7418641b8b3096bae6bdcdd7d7a543d059d7834c72340ee6e2645666"} Oct 30 17:59:52.143037 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:52.143019 4965 generic.go:356] "Generic (PLEG): container finished" podID="59a5a8c6-a647-428d-b181-a0528017fa79" containerID="2ef70466c9ce37dbcfb81a680801db71da93e4deb67b2f092b10b9b9713d66b2" exitCode=0 Oct 30 17:59:52.143356 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:52.143187 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-8498f7756c-zhxg6" event={"ID":"59a5a8c6-a647-428d-b181-a0528017fa79","Type":"ContainerDied","Data":"2ef70466c9ce37dbcfb81a680801db71da93e4deb67b2f092b10b9b9713d66b2"} Oct 30 17:59:52.147813 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:52.147601 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-f4c544ccb-j2kng" Oct 30 17:59:52.197616 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:52.197423 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 17:59:52.197616 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:52.197495 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" podUID="722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 17:59:52.227112 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:52.227083 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9576164-357d-4a63-bedf-f01b139de233-kube-api-access-b9dwc\" (UniqueName: \"kubernetes.io/projected/f9576164-357d-4a63-bedf-f01b139de233-kube-api-access-b9dwc\") pod \"f9576164-357d-4a63-bedf-f01b139de233\" (UID: \"f9576164-357d-4a63-bedf-f01b139de233\") " Oct 30 17:59:52.228304 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:52.228267 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9576164-357d-4a63-bedf-f01b139de233-plugins-config-volume" pod "f9576164-357d-4a63-bedf-f01b139de233" (UID: "f9576164-357d-4a63-bedf-f01b139de233"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:59:52.228478 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:52.228460 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9576164-357d-4a63-bedf-f01b139de233-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9576164-357d-4a63-bedf-f01b139de233-plugins-config-volume\") pod \"f9576164-357d-4a63-bedf-f01b139de233\" (UID: \"f9576164-357d-4a63-bedf-f01b139de233\") " Oct 30 17:59:52.229349 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:52.229329 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9576164-357d-4a63-bedf-f01b139de233-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:59:52.232945 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:52.232800 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9576164-357d-4a63-bedf-f01b139de233-kube-api-access-b9dwc" pod "f9576164-357d-4a63-bedf-f01b139de233" (UID: "f9576164-357d-4a63-bedf-f01b139de233"). InnerVolumeSpecName "kube-api-access-b9dwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:59:52.240778 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:52.239413 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-8498f7756c-zhxg6" Oct 30 17:59:52.330880 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:52.330828 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59a5a8c6-a647-428d-b181-a0528017fa79-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59a5a8c6-a647-428d-b181-a0528017fa79-plugins-config-volume\") pod \"59a5a8c6-a647-428d-b181-a0528017fa79\" (UID: \"59a5a8c6-a647-428d-b181-a0528017fa79\") " Oct 30 17:59:52.331065 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:52.330951 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59a5a8c6-a647-428d-b181-a0528017fa79-kube-api-access-db82c\" (UniqueName: \"kubernetes.io/projected/59a5a8c6-a647-428d-b181-a0528017fa79-kube-api-access-db82c\") pod \"59a5a8c6-a647-428d-b181-a0528017fa79\" (UID: \"59a5a8c6-a647-428d-b181-a0528017fa79\") " Oct 30 17:59:52.331556 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:52.331526 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9dwc\" (UniqueName: \"kubernetes.io/projected/f9576164-357d-4a63-bedf-f01b139de233-kube-api-access-b9dwc\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:59:52.332128 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:52.332102 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a5a8c6-a647-428d-b181-a0528017fa79-plugins-config-volume" pod "59a5a8c6-a647-428d-b181-a0528017fa79" (UID: "59a5a8c6-a647-428d-b181-a0528017fa79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:59:52.334626 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:52.334463 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a5a8c6-a647-428d-b181-a0528017fa79-kube-api-access-db82c" pod "59a5a8c6-a647-428d-b181-a0528017fa79" (UID: "59a5a8c6-a647-428d-b181-a0528017fa79"). InnerVolumeSpecName "kube-api-access-db82c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:59:52.433607 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:52.433521 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59a5a8c6-a647-428d-b181-a0528017fa79-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:59:52.433607 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:52.433552 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db82c\" (UniqueName: \"kubernetes.io/projected/59a5a8c6-a647-428d-b181-a0528017fa79-kube-api-access-db82c\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 17:59:53.153135 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:53.153108 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-8498f7756c-zhxg6" Oct 30 17:59:53.153135 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:53.153125 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-8498f7756c-zhxg6" event={"ID":"59a5a8c6-a647-428d-b181-a0528017fa79","Type":"ContainerDied","Data":"e624655052085d92177f06f4ed98f1bb4b72a9dcb17787e5a2ade956c23ca330"} Oct 30 17:59:53.153458 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:53.153189 4965 scope.go:122] "RemoveContainer" containerID="2ef70466c9ce37dbcfb81a680801db71da93e4deb67b2f092b10b9b9713d66b2" Oct 30 17:59:53.155566 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:53.155544 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" event={"ID":"241c1434-3e5c-4120-8eb5-507bf30a92ec","Type":"ContainerStarted","Data":"da12ea0900c7ab9fa9715f6656827db1d7b795e28eb111c4f0bbfb3b7b3d5b03"} Oct 30 17:59:53.155723 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:53.155707 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" event={"ID":"241c1434-3e5c-4120-8eb5-507bf30a92ec","Type":"ContainerStarted","Data":"ba544a613cdf99bce0e87e927ea0342e6e75f3d92688222727723fdf9c776ff1"} Oct 30 17:59:53.175298 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:53.175122 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-f4c544ccb-j2kng" event={"ID":"f9576164-357d-4a63-bedf-f01b139de233","Type":"ContainerDied","Data":"ef92c74c9de29a594d6991faf5d73bc9fb886b73213eb2c15936bb5611c4e950"} Oct 30 17:59:53.175298 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:53.175259 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-f4c544ccb-j2kng" Oct 30 17:59:53.215072 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:53.214593 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 17:59:53.215072 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:53.214659 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" podUID="722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 17:59:53.224537 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:53.224515 4965 scope.go:122] "RemoveContainer" containerID="9cd3684c7418641b8b3096bae6bdcdd7d7a543d059d7834c72340ee6e2645666" Oct 30 17:59:53.245353 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:53.245297 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" podStartSLOduration=2.245280529 podStartE2EDuration="2.245280529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:59:53.240654857 +0000 UTC m=+2851.292754788" watchObservedRunningTime="2026-10-30 17:59:53.245280529 +0000 UTC m=+2851.297380462" Oct 30 17:59:53.332890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:53.323827 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-8498f7756c-zhxg6"] Oct 30 17:59:53.353890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:53.349636 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fab25f-8498f7756c-zhxg6"] Oct 30 17:59:53.422946 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:53.420933 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fab25f-f4c544ccb-j2kng"] Oct 30 17:59:53.438890 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:53.438842 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fab25f-f4c544ccb-j2kng"] Oct 30 17:59:53.723104 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:53.723012 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 17:59:53.724251 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:53.724220 4965 patch_prober.go:29] interesting pod/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 17:59:53.724400 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:53.724279 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" podUID="241c1434-3e5c-4120-8eb5-507bf30a92ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 17:59:54.116954 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:54.116918 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59a5a8c6-a647-428d-b181-a0528017fa79" path="/var/lib/kubelet/pods/59a5a8c6-a647-428d-b181-a0528017fa79/volumes" Oct 30 17:59:54.117699 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:54.117680 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9576164-357d-4a63-bedf-f01b139de233" path="/var/lib/kubelet/pods/f9576164-357d-4a63-bedf-f01b139de233/volumes" Oct 30 17:59:54.218888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:54.218570 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 17:59:54.218888 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:54.218635 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" podUID="722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 17:59:54.723979 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:54.723945 4965 patch_prober.go:29] interesting pod/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 17:59:54.724359 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:54.724008 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" podUID="241c1434-3e5c-4120-8eb5-507bf30a92ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 17:59:55.191254 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:55.191220 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 30 17:59:55.191452 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:55.191274 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" podUID="722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 30 17:59:55.723796 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:55.723761 4965 patch_prober.go:29] interesting pod/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 17:59:55.724015 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:55.723824 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" podUID="241c1434-3e5c-4120-8eb5-507bf30a92ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 17:59:56.196807 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:56.196770 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:56.197255 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:56.197241 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:56.197887 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:56.197841 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 17:59:56.723673 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:56.723639 4965 patch_prober.go:29] interesting pod/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 17:59:56.723834 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:56.723688 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" podUID="241c1434-3e5c-4120-8eb5-507bf30a92ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 17:59:57.723321 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:57.723288 4965 patch_prober.go:29] interesting pod/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 17:59:57.723751 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:57.723346 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" podUID="241c1434-3e5c-4120-8eb5-507bf30a92ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 17:59:58.833052 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:58.833013 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 17:59:58.834614 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:58.834591 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 17:59:58.839230 ip-10-0-139-222 kubenswrapper[4965]: I1030 17:59:58.838240 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 18:00:00.233026 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.232993 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88"] Oct 30 18:00:00.233858 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.233839 4965 cpu_manager.go:414] "containerMap: removing container" podUID="f9576164-357d-4a63-bedf-f01b139de233" containerName="payload-processing" Oct 30 18:00:00.233858 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.233872 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9576164-357d-4a63-bedf-f01b139de233" containerName="payload-processing" Oct 30 18:00:00.233990 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.233884 4965 cpu_manager.go:414] "containerMap: removing container" podUID="59a5a8c6-a647-428d-b181-a0528017fa79" containerName="payload-pre-processing" Oct 30 18:00:00.233990 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.233892 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59a5a8c6-a647-428d-b181-a0528017fa79" containerName="payload-pre-processing" Oct 30 18:00:00.234076 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.234066 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9576164-357d-4a63-bedf-f01b139de233" containerName="payload-processing" Oct 30 18:00:00.234113 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.234077 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="59a5a8c6-a647-428d-b181-a0528017fa79" containerName="payload-pre-processing" Oct 30 18:00:00.239609 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.239590 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88" Oct 30 18:00:00.261712 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.261549 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88"] Oct 30 18:00:00.267571 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.267533 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\"" Oct 30 18:00:00.268419 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.268394 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz"] Oct 30 18:00:00.276048 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.276028 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj"] Oct 30 18:00:00.276279 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.276243 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz" Oct 30 18:00:00.283119 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.283092 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj" Oct 30 18:00:00.286122 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.286093 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zv75\" (UniqueName: \"kubernetes.io/projected/093a5e7e-2513-47f3-a742-cce2574ac917-kube-api-access-6zv75\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88\" (UID: \"093a5e7e-2513-47f3-a742-cce2574ac917\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88" Oct 30 18:00:00.286256 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.286234 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/093a5e7e-2513-47f3-a742-cce2574ac917-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88\" (UID: \"093a5e7e-2513-47f3-a742-cce2574ac917\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88" Oct 30 18:00:00.296362 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.293820 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz"] Oct 30 18:00:00.307937 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.307905 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj"] Oct 30 18:00:00.333574 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.333541 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw"] Oct 30 18:00:00.343414 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.343385 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw" Oct 30 18:00:00.361386 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.360346 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf"] Oct 30 18:00:00.368376 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.368351 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw"] Oct 30 18:00:00.368514 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.368473 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf" Oct 30 18:00:00.377552 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.377472 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb"] Oct 30 18:00:00.387395 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.387363 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtsnc\" (UniqueName: \"kubernetes.io/projected/63f5bed4-8de4-400c-a622-87e37d050d1c-kube-api-access-wtsnc\") pod \"maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj\" (UID: \"63f5bed4-8de4-400c-a622-87e37d050d1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj" Oct 30 18:00:00.387698 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.387673 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvjrv\" (UniqueName: \"kubernetes.io/projected/35173211-ac01-43fa-885c-b148a4f81ae4-kube-api-access-nvjrv\") pod \"maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf\" (UID: \"35173211-ac01-43fa-885c-b148a4f81ae4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf" Oct 30 18:00:00.387964 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.387943 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zv75\" (UniqueName: \"kubernetes.io/projected/093a5e7e-2513-47f3-a742-cce2574ac917-kube-api-access-6zv75\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88\" (UID: \"093a5e7e-2513-47f3-a742-cce2574ac917\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88" Oct 30 18:00:00.388448 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.388428 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23dc886a-04cf-4d65-b4e5-b9de4c06e228-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz\" (UID: \"23dc886a-04cf-4d65-b4e5-b9de4c06e228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz" Oct 30 18:00:00.389750 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.389727 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e9c7d6c-3ded-495e-809b-5ffccf77b95a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw\" (UID: \"5e9c7d6c-3ded-495e-809b-5ffccf77b95a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw" Oct 30 18:00:00.390456 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.390437 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35173211-ac01-43fa-885c-b148a4f81ae4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf\" (UID: \"35173211-ac01-43fa-885c-b148a4f81ae4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf" Oct 30 18:00:00.390693 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.390672 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/093a5e7e-2513-47f3-a742-cce2574ac917-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88\" (UID: \"093a5e7e-2513-47f3-a742-cce2574ac917\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88" Oct 30 18:00:00.392221 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.392202 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jgqc\" (UniqueName: \"kubernetes.io/projected/23dc886a-04cf-4d65-b4e5-b9de4c06e228-kube-api-access-8jgqc\") pod \"maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz\" (UID: \"23dc886a-04cf-4d65-b4e5-b9de4c06e228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz" Oct 30 18:00:00.392505 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.392483 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63f5bed4-8de4-400c-a622-87e37d050d1c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj\" (UID: \"63f5bed4-8de4-400c-a622-87e37d050d1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj" Oct 30 18:00:00.392727 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.392707 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22thg\" (UniqueName: \"kubernetes.io/projected/5e9c7d6c-3ded-495e-809b-5ffccf77b95a-kube-api-access-22thg\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw\" (UID: \"5e9c7d6c-3ded-495e-809b-5ffccf77b95a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw" Oct 30 18:00:00.393278 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.392082 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/093a5e7e-2513-47f3-a742-cce2574ac917-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88\" (UID: \"093a5e7e-2513-47f3-a742-cce2574ac917\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88" Oct 30 18:00:00.403363 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.403345 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf"] Oct 30 18:00:00.403539 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.403461 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb" Oct 30 18:00:00.403780 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.403765 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp"] Oct 30 18:00:00.412110 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.411993 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb"] Oct 30 18:00:00.412543 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.412084 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp" Oct 30 18:00:00.422285 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.422261 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zv75\" (UniqueName: \"kubernetes.io/projected/093a5e7e-2513-47f3-a742-cce2574ac917-kube-api-access-6zv75\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88\" (UID: \"093a5e7e-2513-47f3-a742-cce2574ac917\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88" Oct 30 18:00:00.426054 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.423213 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp"] Oct 30 18:00:00.494272 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.494178 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22thg\" (UniqueName: \"kubernetes.io/projected/5e9c7d6c-3ded-495e-809b-5ffccf77b95a-kube-api-access-22thg\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw\" (UID: \"5e9c7d6c-3ded-495e-809b-5ffccf77b95a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw" Oct 30 18:00:00.494471 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.494285 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtsnc\" (UniqueName: \"kubernetes.io/projected/63f5bed4-8de4-400c-a622-87e37d050d1c-kube-api-access-wtsnc\") pod \"maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj\" (UID: \"63f5bed4-8de4-400c-a622-87e37d050d1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj" Oct 30 18:00:00.494471 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.494333 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvjrv\" (UniqueName: \"kubernetes.io/projected/35173211-ac01-43fa-885c-b148a4f81ae4-kube-api-access-nvjrv\") pod \"maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf\" (UID: \"35173211-ac01-43fa-885c-b148a4f81ae4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf" Oct 30 18:00:00.494471 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.494362 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr8zx\" (UniqueName: \"kubernetes.io/projected/02e21c00-730d-41ec-a53d-ea9f556c2459-kube-api-access-nr8zx\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp\" (UID: \"02e21c00-730d-41ec-a53d-ea9f556c2459\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp" Oct 30 18:00:00.494641 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.494527 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23dc886a-04cf-4d65-b4e5-b9de4c06e228-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz\" (UID: \"23dc886a-04cf-4d65-b4e5-b9de4c06e228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz" Oct 30 18:00:00.494641 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.494599 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e9c7d6c-3ded-495e-809b-5ffccf77b95a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw\" (UID: \"5e9c7d6c-3ded-495e-809b-5ffccf77b95a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw" Oct 30 18:00:00.494641 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.494628 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4abe15c2-d8d1-40aa-a0d2-90673380e3ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb\" (UID: \"4abe15c2-d8d1-40aa-a0d2-90673380e3ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb" Oct 30 18:00:00.494850 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.494655 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7898p\" (UniqueName: \"kubernetes.io/projected/4abe15c2-d8d1-40aa-a0d2-90673380e3ef-kube-api-access-7898p\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb\" (UID: \"4abe15c2-d8d1-40aa-a0d2-90673380e3ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb" Oct 30 18:00:00.494850 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.494741 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35173211-ac01-43fa-885c-b148a4f81ae4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf\" (UID: \"35173211-ac01-43fa-885c-b148a4f81ae4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf" Oct 30 18:00:00.494850 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.494778 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02e21c00-730d-41ec-a53d-ea9f556c2459-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp\" (UID: \"02e21c00-730d-41ec-a53d-ea9f556c2459\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp" Oct 30 18:00:00.495044 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.494950 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jgqc\" (UniqueName: \"kubernetes.io/projected/23dc886a-04cf-4d65-b4e5-b9de4c06e228-kube-api-access-8jgqc\") pod \"maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz\" (UID: \"23dc886a-04cf-4d65-b4e5-b9de4c06e228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz" Oct 30 18:00:00.495044 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.495008 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63f5bed4-8de4-400c-a622-87e37d050d1c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj\" (UID: \"63f5bed4-8de4-400c-a622-87e37d050d1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj" Oct 30 18:00:00.495197 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.495177 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23dc886a-04cf-4d65-b4e5-b9de4c06e228-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz\" (UID: \"23dc886a-04cf-4d65-b4e5-b9de4c06e228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz" Oct 30 18:00:00.495404 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.495381 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e9c7d6c-3ded-495e-809b-5ffccf77b95a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw\" (UID: \"5e9c7d6c-3ded-495e-809b-5ffccf77b95a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw" Oct 30 18:00:00.495725 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.495704 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35173211-ac01-43fa-885c-b148a4f81ae4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf\" (UID: \"35173211-ac01-43fa-885c-b148a4f81ae4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf" Oct 30 18:00:00.495803 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.495782 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63f5bed4-8de4-400c-a622-87e37d050d1c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj\" (UID: \"63f5bed4-8de4-400c-a622-87e37d050d1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj" Oct 30 18:00:00.517961 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.517934 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtsnc\" (UniqueName: \"kubernetes.io/projected/63f5bed4-8de4-400c-a622-87e37d050d1c-kube-api-access-wtsnc\") pod \"maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj\" (UID: \"63f5bed4-8de4-400c-a622-87e37d050d1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj" Oct 30 18:00:00.518065 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.518029 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jgqc\" (UniqueName: \"kubernetes.io/projected/23dc886a-04cf-4d65-b4e5-b9de4c06e228-kube-api-access-8jgqc\") pod \"maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz\" (UID: \"23dc886a-04cf-4d65-b4e5-b9de4c06e228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz" Oct 30 18:00:00.518168 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.518148 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvjrv\" (UniqueName: \"kubernetes.io/projected/35173211-ac01-43fa-885c-b148a4f81ae4-kube-api-access-nvjrv\") pod \"maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf\" (UID: \"35173211-ac01-43fa-885c-b148a4f81ae4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf" Oct 30 18:00:00.518361 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.518345 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22thg\" (UniqueName: \"kubernetes.io/projected/5e9c7d6c-3ded-495e-809b-5ffccf77b95a-kube-api-access-22thg\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw\" (UID: \"5e9c7d6c-3ded-495e-809b-5ffccf77b95a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw" Oct 30 18:00:00.563121 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.563092 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88" Oct 30 18:00:00.639146 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.609060 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr8zx\" (UniqueName: \"kubernetes.io/projected/02e21c00-730d-41ec-a53d-ea9f556c2459-kube-api-access-nr8zx\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp\" (UID: \"02e21c00-730d-41ec-a53d-ea9f556c2459\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp" Oct 30 18:00:00.647299 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.640912 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj" Oct 30 18:00:00.648919 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.644940 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz" Oct 30 18:00:00.648919 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.647634 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4abe15c2-d8d1-40aa-a0d2-90673380e3ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb\" (UID: \"4abe15c2-d8d1-40aa-a0d2-90673380e3ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb" Oct 30 18:00:00.648919 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.647674 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7898p\" (UniqueName: \"kubernetes.io/projected/4abe15c2-d8d1-40aa-a0d2-90673380e3ef-kube-api-access-7898p\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb\" (UID: \"4abe15c2-d8d1-40aa-a0d2-90673380e3ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb" Oct 30 18:00:00.648919 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.647713 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02e21c00-730d-41ec-a53d-ea9f556c2459-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp\" (UID: \"02e21c00-730d-41ec-a53d-ea9f556c2459\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp" Oct 30 18:00:00.648919 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.648785 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4abe15c2-d8d1-40aa-a0d2-90673380e3ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb\" (UID: \"4abe15c2-d8d1-40aa-a0d2-90673380e3ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb" Oct 30 18:00:00.648919 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.648846 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02e21c00-730d-41ec-a53d-ea9f556c2459-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp\" (UID: \"02e21c00-730d-41ec-a53d-ea9f556c2459\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp" Oct 30 18:00:00.671767 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.671736 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr8zx\" (UniqueName: \"kubernetes.io/projected/02e21c00-730d-41ec-a53d-ea9f556c2459-kube-api-access-nr8zx\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp\" (UID: \"02e21c00-730d-41ec-a53d-ea9f556c2459\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp" Oct 30 18:00:00.672601 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.672558 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw" Oct 30 18:00:00.674531 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.674496 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7898p\" (UniqueName: \"kubernetes.io/projected/4abe15c2-d8d1-40aa-a0d2-90673380e3ef-kube-api-access-7898p\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb\" (UID: \"4abe15c2-d8d1-40aa-a0d2-90673380e3ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb" Oct 30 18:00:00.708468 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.708219 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf" Oct 30 18:00:00.726364 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.726308 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb" Oct 30 18:00:00.751692 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:00.751659 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp" Oct 30 18:00:01.146074 ip-10-0-139-222 kubenswrapper[4965]: W1030 18:00:01.146038 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod093a5e7e_2513_47f3_a742_cce2574ac917.slice/crio-57bfa982f991e0be11f9035e5f717c62baa18d7b5aa3472d19f8629cfd20ac56 WatchSource:0}: Error finding container 57bfa982f991e0be11f9035e5f717c62baa18d7b5aa3472d19f8629cfd20ac56: Status 404 returned error can't find the container with id 57bfa982f991e0be11f9035e5f717c62baa18d7b5aa3472d19f8629cfd20ac56 Oct 30 18:00:01.146328 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:01.146303 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88"] Oct 30 18:00:01.231226 ip-10-0-139-222 kubenswrapper[4965]: W1030 18:00:01.231195 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f5bed4_8de4_400c_a622_87e37d050d1c.slice/crio-8c23cd9ce0c0d21171b2936829b6db8ce13bb1ec07bd62cefe7ca2ff52d25a3e WatchSource:0}: Error finding container 8c23cd9ce0c0d21171b2936829b6db8ce13bb1ec07bd62cefe7ca2ff52d25a3e: Status 404 returned error can't find the container with id 8c23cd9ce0c0d21171b2936829b6db8ce13bb1ec07bd62cefe7ca2ff52d25a3e Oct 30 18:00:01.232450 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:01.232425 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj"] Oct 30 18:00:01.273924 ip-10-0-139-222 kubenswrapper[4965]: W1030 18:00:01.273837 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e9c7d6c_3ded_495e_809b_5ffccf77b95a.slice/crio-38b4b69ae371d8f7fbee83f92cb206663ed5d0c3a73bb72fe27d23015323d9a8 WatchSource:0}: Error finding container 38b4b69ae371d8f7fbee83f92cb206663ed5d0c3a73bb72fe27d23015323d9a8: Status 404 returned error can't find the container with id 38b4b69ae371d8f7fbee83f92cb206663ed5d0c3a73bb72fe27d23015323d9a8 Oct 30 18:00:01.278318 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:01.278296 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw"] Oct 30 18:00:01.287988 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:01.287962 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb"] Oct 30 18:00:01.295698 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:01.295675 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp"] Oct 30 18:00:01.304408 ip-10-0-139-222 kubenswrapper[4965]: W1030 18:00:01.304370 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23dc886a_04cf_4d65_b4e5_b9de4c06e228.slice/crio-002c3d72e7ad42a49c76582180605559e51bd4e33ec2593031c9df44d69e0b06 WatchSource:0}: Error finding container 002c3d72e7ad42a49c76582180605559e51bd4e33ec2593031c9df44d69e0b06: Status 404 returned error can't find the container with id 002c3d72e7ad42a49c76582180605559e51bd4e33ec2593031c9df44d69e0b06 Oct 30 18:00:01.304786 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:01.304758 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88" event={"ID":"093a5e7e-2513-47f3-a742-cce2574ac917","Type":"ContainerStarted","Data":"57bfa982f991e0be11f9035e5f717c62baa18d7b5aa3472d19f8629cfd20ac56"} Oct 30 18:00:01.306957 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:01.306897 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw" event={"ID":"5e9c7d6c-3ded-495e-809b-5ffccf77b95a","Type":"ContainerStarted","Data":"38b4b69ae371d8f7fbee83f92cb206663ed5d0c3a73bb72fe27d23015323d9a8"} Oct 30 18:00:01.308330 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:01.308311 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj" event={"ID":"63f5bed4-8de4-400c-a622-87e37d050d1c","Type":"ContainerStarted","Data":"8c23cd9ce0c0d21171b2936829b6db8ce13bb1ec07bd62cefe7ca2ff52d25a3e"} Oct 30 18:00:01.310784 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:01.310763 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz"] Oct 30 18:00:01.405042 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:01.405013 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf"] Oct 30 18:00:01.562072 ip-10-0-139-222 kubenswrapper[4965]: W1030 18:00:01.561985 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35173211_ac01_43fa_885c_b148a4f81ae4.slice/crio-c9e68dc1a9077152f22b6f8b346a8b5a2cea9f7bfbdedb0ee2e0cc5e56e64a51 WatchSource:0}: Error finding container c9e68dc1a9077152f22b6f8b346a8b5a2cea9f7bfbdedb0ee2e0cc5e56e64a51: Status 404 returned error can't find the container with id c9e68dc1a9077152f22b6f8b346a8b5a2cea9f7bfbdedb0ee2e0cc5e56e64a51 Oct 30 18:00:02.317795 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:02.317759 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb" event={"ID":"4abe15c2-d8d1-40aa-a0d2-90673380e3ef","Type":"ContainerStarted","Data":"066b41e0a642a0d06591fe4bbe672a42650e168513c51562d14e33626bafdd2b"} Oct 30 18:00:02.318244 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:02.317822 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb" event={"ID":"4abe15c2-d8d1-40aa-a0d2-90673380e3ef","Type":"ContainerStarted","Data":"b7b8c6bc62c7c5e43e633db0517192c8bab1358bd556a4f7de1467b40df37569"} Oct 30 18:00:02.319426 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:02.319399 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp" event={"ID":"02e21c00-730d-41ec-a53d-ea9f556c2459","Type":"ContainerStarted","Data":"f6668b5e9077da189c3c24509de0b79358de39facee68199097f191bbb81b79d"} Oct 30 18:00:02.319551 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:02.319451 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp" event={"ID":"02e21c00-730d-41ec-a53d-ea9f556c2459","Type":"ContainerStarted","Data":"8885466fc7eeed3f68cb1ac364357c941591ea4ef8af01f1cd437e619c78c6e9"} Oct 30 18:00:02.320943 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:02.320916 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz" event={"ID":"23dc886a-04cf-4d65-b4e5-b9de4c06e228","Type":"ContainerStarted","Data":"e2bc858f1b5003f65bdc9de24fc933b87867cc11139b7170a98a53d944b40549"} Oct 30 18:00:02.321078 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:02.320963 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz" event={"ID":"23dc886a-04cf-4d65-b4e5-b9de4c06e228","Type":"ContainerStarted","Data":"002c3d72e7ad42a49c76582180605559e51bd4e33ec2593031c9df44d69e0b06"} Oct 30 18:00:02.322500 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:02.322472 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88" event={"ID":"093a5e7e-2513-47f3-a742-cce2574ac917","Type":"ContainerStarted","Data":"e6d1adff98b4ab927584fa2a5468b608dea63cfe9c62b44eca00fca87ca71615"} Oct 30 18:00:02.323974 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:02.323950 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw" event={"ID":"5e9c7d6c-3ded-495e-809b-5ffccf77b95a","Type":"ContainerStarted","Data":"4aa281fb24788149960c27fbeb73cb1a88595bdb01025c56ca3b89df4cdc836b"} Oct 30 18:00:02.325813 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:02.325789 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj" event={"ID":"63f5bed4-8de4-400c-a622-87e37d050d1c","Type":"ContainerStarted","Data":"db91b63ad5bd643de73a3fca877f86969a348dfc990806f437b379d1f7490c39"} Oct 30 18:00:02.327436 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:02.327413 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf" event={"ID":"35173211-ac01-43fa-885c-b148a4f81ae4","Type":"ContainerStarted","Data":"29639a11c34ee6847aec413cd7eb2a81cef7002528c0080ca04414b5a813df4e"} Oct 30 18:00:02.327538 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:02.327454 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf" event={"ID":"35173211-ac01-43fa-885c-b148a4f81ae4","Type":"ContainerStarted","Data":"c9e68dc1a9077152f22b6f8b346a8b5a2cea9f7bfbdedb0ee2e0cc5e56e64a51"} Oct 30 18:00:02.498885 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:02.487236 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88" podStartSLOduration=2.487216862 podStartE2EDuration="2.487216862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:02.486232602 +0000 UTC m=+2860.538332557" watchObservedRunningTime="2026-10-30 18:00:02.487216862 +0000 UTC m=+2860.539316794" Oct 30 18:00:02.532935 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:02.514372 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb" podStartSLOduration=2.514342298 podStartE2EDuration="2.514342298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:02.389164584 +0000 UTC m=+2860.441264516" watchObservedRunningTime="2026-10-30 18:00:02.514342298 +0000 UTC m=+2860.566442296" Oct 30 18:00:02.559019 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:02.558962 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-455e4a-6d96cfd86b-kmrsv"] Oct 30 18:00:02.613933 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:02.613320 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-6d96cfd86b-kmrsv" Oct 30 18:00:02.629138 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:02.628459 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-455e4a-dockercfg-zslsv\"" Oct 30 18:00:02.629824 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:02.629326 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-455e4a\"" Oct 30 18:00:02.635768 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:02.635716 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw" podStartSLOduration=2.635701877 podStartE2EDuration="2.635701877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:02.609032039 +0000 UTC m=+2860.661131970" watchObservedRunningTime="2026-10-30 18:00:02.635701877 +0000 UTC m=+2860.687801857" Oct 30 18:00:02.644525 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:02.644499 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-455e4a-6d96cfd86b-kmrsv"] Oct 30 18:00:02.695152 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:02.695089 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz" podStartSLOduration=2.695070349 podStartE2EDuration="2.695070349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:02.652059565 +0000 UTC m=+2860.704159497" watchObservedRunningTime="2026-10-30 18:00:02.695070349 +0000 UTC m=+2860.747170281" Oct 30 18:00:02.695460 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:02.695431 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp" podStartSLOduration=2.695419886 podStartE2EDuration="2.695419886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:02.691291908 +0000 UTC m=+2860.743391849" watchObservedRunningTime="2026-10-30 18:00:02.695419886 +0000 UTC m=+2860.747519818" Oct 30 18:00:02.728523 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:02.728495 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mtjf\" (UniqueName: \"kubernetes.io/projected/4bd72d29-e1f9-49cd-8388-156d3796563a-kube-api-access-2mtjf\") pod \"payload-processing-e2e-ipp-a-455e4a-6d96cfd86b-kmrsv\" (UID: \"4bd72d29-e1f9-49cd-8388-156d3796563a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-6d96cfd86b-kmrsv" Oct 30 18:00:02.728740 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:02.728711 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd72d29-e1f9-49cd-8388-156d3796563a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-455e4a-6d96cfd86b-kmrsv\" (UID: \"4bd72d29-e1f9-49cd-8388-156d3796563a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-6d96cfd86b-kmrsv" Oct 30 18:00:02.739062 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:02.739016 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj" podStartSLOduration=2.739002554 podStartE2EDuration="2.739002554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:02.734625677 +0000 UTC m=+2860.786725610" watchObservedRunningTime="2026-10-30 18:00:02.739002554 +0000 UTC m=+2860.791102485" Oct 30 18:00:02.774965 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:02.774913 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf" podStartSLOduration=2.774900448 podStartE2EDuration="2.774900448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:02.773556619 +0000 UTC m=+2860.825656551" watchObservedRunningTime="2026-10-30 18:00:02.774900448 +0000 UTC m=+2860.827000380" Oct 30 18:00:02.822145 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:02.822109 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd" podUID="a2f007f6-27a0-4cfb-ba7e-c20f030397f1" containerName="main" containerID="cri-o://06fa90304cb5bdb5c5fc146fa6f7e1a8a91180a71ecb10209dd7a70df739e6f1" gracePeriod=45 Oct 30 18:00:02.829841 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:02.829813 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mtjf\" (UniqueName: \"kubernetes.io/projected/4bd72d29-e1f9-49cd-8388-156d3796563a-kube-api-access-2mtjf\") pod \"payload-processing-e2e-ipp-a-455e4a-6d96cfd86b-kmrsv\" (UID: \"4bd72d29-e1f9-49cd-8388-156d3796563a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-6d96cfd86b-kmrsv" Oct 30 18:00:02.830047 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:02.830027 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd72d29-e1f9-49cd-8388-156d3796563a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-455e4a-6d96cfd86b-kmrsv\" (UID: \"4bd72d29-e1f9-49cd-8388-156d3796563a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-6d96cfd86b-kmrsv" Oct 30 18:00:02.831147 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:02.831122 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd72d29-e1f9-49cd-8388-156d3796563a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-455e4a-6d96cfd86b-kmrsv\" (UID: \"4bd72d29-e1f9-49cd-8388-156d3796563a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-6d96cfd86b-kmrsv" Oct 30 18:00:02.853834 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:02.853805 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mtjf\" (UniqueName: \"kubernetes.io/projected/4bd72d29-e1f9-49cd-8388-156d3796563a-kube-api-access-2mtjf\") pod \"payload-processing-e2e-ipp-a-455e4a-6d96cfd86b-kmrsv\" (UID: \"4bd72d29-e1f9-49cd-8388-156d3796563a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-6d96cfd86b-kmrsv" Oct 30 18:00:02.892723 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:00:02.892695 4965 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2f007f6_27a0_4cfb_ba7e_c20f030397f1.slice/crio-06fa90304cb5bdb5c5fc146fa6f7e1a8a91180a71ecb10209dd7a70df739e6f1.scope\": RecentStats: unable to find data in memory cache]" Oct 30 18:00:03.000996 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.000970 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-6d96cfd86b-kmrsv" Oct 30 18:00:03.123471 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.123442 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd" Oct 30 18:00:03.244380 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.244283 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-kube-api-access-mtvt5\" (UniqueName: \"kubernetes.io/projected/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-kube-api-access-mtvt5\") pod \"a2f007f6-27a0-4cfb-ba7e-c20f030397f1\" (UID: \"a2f007f6-27a0-4cfb-ba7e-c20f030397f1\") " Oct 30 18:00:03.244380 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.244364 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-model-cache\") pod \"a2f007f6-27a0-4cfb-ba7e-c20f030397f1\" (UID: \"a2f007f6-27a0-4cfb-ba7e-c20f030397f1\") " Oct 30 18:00:03.244605 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.244559 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-tls-certs\") pod \"a2f007f6-27a0-4cfb-ba7e-c20f030397f1\" (UID: \"a2f007f6-27a0-4cfb-ba7e-c20f030397f1\") " Oct 30 18:00:03.244660 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.244615 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-model-cache" pod "a2f007f6-27a0-4cfb-ba7e-c20f030397f1" (UID: "a2f007f6-27a0-4cfb-ba7e-c20f030397f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:00:03.245047 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.245023 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-tmp-dir\") pod \"a2f007f6-27a0-4cfb-ba7e-c20f030397f1\" (UID: \"a2f007f6-27a0-4cfb-ba7e-c20f030397f1\") " Oct 30 18:00:03.245369 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.245315 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-dshm\") pod \"a2f007f6-27a0-4cfb-ba7e-c20f030397f1\" (UID: \"a2f007f6-27a0-4cfb-ba7e-c20f030397f1\") " Oct 30 18:00:03.245549 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.245531 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-home\" (UniqueName: \"kubernetes.io/empty-dir/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-home\") pod \"a2f007f6-27a0-4cfb-ba7e-c20f030397f1\" (UID: \"a2f007f6-27a0-4cfb-ba7e-c20f030397f1\") " Oct 30 18:00:03.245912 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.245398 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-tmp-dir" pod "a2f007f6-27a0-4cfb-ba7e-c20f030397f1" (UID: "a2f007f6-27a0-4cfb-ba7e-c20f030397f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:00:03.245998 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.245793 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-home" pod "a2f007f6-27a0-4cfb-ba7e-c20f030397f1" (UID: "a2f007f6-27a0-4cfb-ba7e-c20f030397f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:00:03.246474 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.246450 4965 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-model-cache\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:00:03.246591 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.246475 4965 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-tmp-dir\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:00:03.246591 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.246488 4965 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-home\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:00:03.247016 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.246995 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-tls-certs" pod "a2f007f6-27a0-4cfb-ba7e-c20f030397f1" (UID: "a2f007f6-27a0-4cfb-ba7e-c20f030397f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:00:03.247117 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.247067 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-kube-api-access-mtvt5" pod "a2f007f6-27a0-4cfb-ba7e-c20f030397f1" (UID: "a2f007f6-27a0-4cfb-ba7e-c20f030397f1"). InnerVolumeSpecName "kube-api-access-mtvt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:00:03.248327 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.248305 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-dshm" pod "a2f007f6-27a0-4cfb-ba7e-c20f030397f1" (UID: "a2f007f6-27a0-4cfb-ba7e-c20f030397f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:00:03.335433 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.335400 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-455e4a-6d96cfd86b-kmrsv"] Oct 30 18:00:03.343673 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.343641 4965 generic.go:356] "Generic (PLEG): container finished" podID="02e21c00-730d-41ec-a53d-ea9f556c2459" containerID="f6668b5e9077da189c3c24509de0b79358de39facee68199097f191bbb81b79d" exitCode=60 Oct 30 18:00:03.343847 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.343686 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp" event={"ID":"02e21c00-730d-41ec-a53d-ea9f556c2459","Type":"ContainerDied","Data":"f6668b5e9077da189c3c24509de0b79358de39facee68199097f191bbb81b79d"} Oct 30 18:00:03.344170 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.344150 4965 scope.go:122] "RemoveContainer" containerID="f6668b5e9077da189c3c24509de0b79358de39facee68199097f191bbb81b79d" Oct 30 18:00:03.348684 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.348644 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtvt5\" (UniqueName: \"kubernetes.io/projected/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-kube-api-access-mtvt5\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:00:03.348911 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.348890 4965 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-tls-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:00:03.349073 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.349060 4965 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2f007f6-27a0-4cfb-ba7e-c20f030397f1-dshm\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:00:03.354245 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.351412 4965 generic.go:356] "Generic (PLEG): container finished" podID="23dc886a-04cf-4d65-b4e5-b9de4c06e228" containerID="e2bc858f1b5003f65bdc9de24fc933b87867cc11139b7170a98a53d944b40549" exitCode=60 Oct 30 18:00:03.354245 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.351530 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz" event={"ID":"23dc886a-04cf-4d65-b4e5-b9de4c06e228","Type":"ContainerDied","Data":"e2bc858f1b5003f65bdc9de24fc933b87867cc11139b7170a98a53d944b40549"} Oct 30 18:00:03.354245 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.351855 4965 scope.go:122] "RemoveContainer" containerID="e2bc858f1b5003f65bdc9de24fc933b87867cc11139b7170a98a53d944b40549" Oct 30 18:00:03.354245 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.354234 4965 generic.go:356] "Generic (PLEG): container finished" podID="093a5e7e-2513-47f3-a742-cce2574ac917" containerID="e6d1adff98b4ab927584fa2a5468b608dea63cfe9c62b44eca00fca87ca71615" exitCode=60 Oct 30 18:00:03.354488 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.354333 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88" event={"ID":"093a5e7e-2513-47f3-a742-cce2574ac917","Type":"ContainerDied","Data":"e6d1adff98b4ab927584fa2a5468b608dea63cfe9c62b44eca00fca87ca71615"} Oct 30 18:00:03.354583 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.354563 4965 scope.go:122] "RemoveContainer" containerID="e6d1adff98b4ab927584fa2a5468b608dea63cfe9c62b44eca00fca87ca71615" Oct 30 18:00:03.390785 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.390734 4965 generic.go:356] "Generic (PLEG): container finished" podID="a2f007f6-27a0-4cfb-ba7e-c20f030397f1" containerID="06fa90304cb5bdb5c5fc146fa6f7e1a8a91180a71ecb10209dd7a70df739e6f1" exitCode=0 Oct 30 18:00:03.391474 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.391147 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd" Oct 30 18:00:03.392240 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.391177 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd" event={"ID":"a2f007f6-27a0-4cfb-ba7e-c20f030397f1","Type":"ContainerDied","Data":"06fa90304cb5bdb5c5fc146fa6f7e1a8a91180a71ecb10209dd7a70df739e6f1"} Oct 30 18:00:03.392919 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.392893 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd" event={"ID":"a2f007f6-27a0-4cfb-ba7e-c20f030397f1","Type":"ContainerDied","Data":"342f9a1cb7fe04aefe1081caecabc795859c354aed91b4cfc64325a7097d87dd"} Oct 30 18:00:03.393093 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.393047 4965 scope.go:122] "RemoveContainer" containerID="06fa90304cb5bdb5c5fc146fa6f7e1a8a91180a71ecb10209dd7a70df739e6f1" Oct 30 18:00:03.398897 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.398370 4965 generic.go:356] "Generic (PLEG): container finished" podID="5e9c7d6c-3ded-495e-809b-5ffccf77b95a" containerID="4aa281fb24788149960c27fbeb73cb1a88595bdb01025c56ca3b89df4cdc836b" exitCode=60 Oct 30 18:00:03.398897 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.398442 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw" event={"ID":"5e9c7d6c-3ded-495e-809b-5ffccf77b95a","Type":"ContainerDied","Data":"4aa281fb24788149960c27fbeb73cb1a88595bdb01025c56ca3b89df4cdc836b"} Oct 30 18:00:03.399062 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.398956 4965 scope.go:122] "RemoveContainer" containerID="4aa281fb24788149960c27fbeb73cb1a88595bdb01025c56ca3b89df4cdc836b" Oct 30 18:00:03.402600 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.402580 4965 generic.go:356] "Generic (PLEG): container finished" podID="63f5bed4-8de4-400c-a622-87e37d050d1c" containerID="db91b63ad5bd643de73a3fca877f86969a348dfc990806f437b379d1f7490c39" exitCode=60 Oct 30 18:00:03.403331 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.403297 4965 scope.go:122] "RemoveContainer" containerID="db91b63ad5bd643de73a3fca877f86969a348dfc990806f437b379d1f7490c39" Oct 30 18:00:03.403677 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.403655 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj" event={"ID":"63f5bed4-8de4-400c-a622-87e37d050d1c","Type":"ContainerDied","Data":"db91b63ad5bd643de73a3fca877f86969a348dfc990806f437b379d1f7490c39"} Oct 30 18:00:03.406688 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.406642 4965 generic.go:356] "Generic (PLEG): container finished" podID="35173211-ac01-43fa-885c-b148a4f81ae4" containerID="29639a11c34ee6847aec413cd7eb2a81cef7002528c0080ca04414b5a813df4e" exitCode=60 Oct 30 18:00:03.406785 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.406757 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf" event={"ID":"35173211-ac01-43fa-885c-b148a4f81ae4","Type":"ContainerDied","Data":"29639a11c34ee6847aec413cd7eb2a81cef7002528c0080ca04414b5a813df4e"} Oct 30 18:00:03.407209 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.407136 4965 scope.go:122] "RemoveContainer" containerID="29639a11c34ee6847aec413cd7eb2a81cef7002528c0080ca04414b5a813df4e" Oct 30 18:00:03.408973 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.408903 4965 generic.go:356] "Generic (PLEG): container finished" podID="4abe15c2-d8d1-40aa-a0d2-90673380e3ef" containerID="066b41e0a642a0d06591fe4bbe672a42650e168513c51562d14e33626bafdd2b" exitCode=60 Oct 30 18:00:03.408973 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.408949 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb" event={"ID":"4abe15c2-d8d1-40aa-a0d2-90673380e3ef","Type":"ContainerDied","Data":"066b41e0a642a0d06591fe4bbe672a42650e168513c51562d14e33626bafdd2b"} Oct 30 18:00:03.409624 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.409329 4965 scope.go:122] "RemoveContainer" containerID="066b41e0a642a0d06591fe4bbe672a42650e168513c51562d14e33626bafdd2b" Oct 30 18:00:03.418663 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.418610 4965 scope.go:122] "RemoveContainer" containerID="06fa90304cb5bdb5c5fc146fa6f7e1a8a91180a71ecb10209dd7a70df739e6f1" Oct 30 18:00:03.418998 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:00:03.418976 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06fa90304cb5bdb5c5fc146fa6f7e1a8a91180a71ecb10209dd7a70df739e6f1\": container with ID starting with 06fa90304cb5bdb5c5fc146fa6f7e1a8a91180a71ecb10209dd7a70df739e6f1 not found: ID does not exist" containerID="06fa90304cb5bdb5c5fc146fa6f7e1a8a91180a71ecb10209dd7a70df739e6f1" Oct 30 18:00:03.419092 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.419008 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06fa90304cb5bdb5c5fc146fa6f7e1a8a91180a71ecb10209dd7a70df739e6f1"} err="failed to get container status \"06fa90304cb5bdb5c5fc146fa6f7e1a8a91180a71ecb10209dd7a70df739e6f1\": rpc error: code = NotFound desc = could not find container \"06fa90304cb5bdb5c5fc146fa6f7e1a8a91180a71ecb10209dd7a70df739e6f1\": container with ID starting with 06fa90304cb5bdb5c5fc146fa6f7e1a8a91180a71ecb10209dd7a70df739e6f1 not found: ID does not exist" Oct 30 18:00:03.469371 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.469343 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd"] Oct 30 18:00:03.476628 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:03.476264 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-c31c73/sub-test-model-c31c73-kserve-6cdfb75949-vwxkd"] Oct 30 18:00:04.103768 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:04.103731 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2f007f6-27a0-4cfb-ba7e-c20f030397f1" path="/var/lib/kubelet/pods/a2f007f6-27a0-4cfb-ba7e-c20f030397f1/volumes" Oct 30 18:00:04.419438 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:04.419397 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz" event={"ID":"23dc886a-04cf-4d65-b4e5-b9de4c06e228","Type":"ContainerStarted","Data":"3a1c9bc836aadcc024c5a74616d0cd44ba3291e16cdb4fc48932f2ca7929c7d3"} Oct 30 18:00:04.421420 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:04.421394 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88" event={"ID":"093a5e7e-2513-47f3-a742-cce2574ac917","Type":"ContainerStarted","Data":"ef1efc85355a3000742912381ae9a1723c8824b127781d81f989efbf43dc7156"} Oct 30 18:00:04.424175 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:04.424154 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw" event={"ID":"5e9c7d6c-3ded-495e-809b-5ffccf77b95a","Type":"ContainerStarted","Data":"c22612339c3c684db5c489e1ec9c33fbd113d367d673dd7e8fe00e3e49142923"} Oct 30 18:00:04.426074 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:04.426052 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj" event={"ID":"63f5bed4-8de4-400c-a622-87e37d050d1c","Type":"ContainerStarted","Data":"f4a3f6f231ccf18d3802fc96319127b011a289ddafff99b956c0fc6b8bfd788a"} Oct 30 18:00:04.427913 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:04.427889 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf" event={"ID":"35173211-ac01-43fa-885c-b148a4f81ae4","Type":"ContainerStarted","Data":"a5fbb0b7c22e0cc229b1e28c15b9ab08f2d4fd158d0d153e2ad6ccc30b203c23"} Oct 30 18:00:04.434119 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:04.434099 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb" event={"ID":"4abe15c2-d8d1-40aa-a0d2-90673380e3ef","Type":"ContainerStarted","Data":"01cbc2400a46a536a49513cb378cfd59427af46632bf2aee158e64f5e1c4b646"} Oct 30 18:00:04.436248 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:04.436227 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp" event={"ID":"02e21c00-730d-41ec-a53d-ea9f556c2459","Type":"ContainerStarted","Data":"d20474cdf0a35bcc396344e967bd5d2002b7aa318b9694c4a42fe37793ca545d"} Oct 30 18:00:04.439074 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:04.439053 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-6d96cfd86b-kmrsv" event={"ID":"4bd72d29-e1f9-49cd-8388-156d3796563a","Type":"ContainerStarted","Data":"bcc554a350e0719015bd25b874f26c48236a8b484aef03259a575bc0dee24e1f"} Oct 30 18:00:04.439192 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:04.439101 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-6d96cfd86b-kmrsv" Oct 30 18:00:04.439192 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:04.439127 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-6d96cfd86b-kmrsv" event={"ID":"4bd72d29-e1f9-49cd-8388-156d3796563a","Type":"ContainerStarted","Data":"51844622ee9814b9f6ab39e5c549f99c4093565a6a642b8126c64910a67d4125"} Oct 30 18:00:04.720235 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:04.720187 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-6d96cfd86b-kmrsv" podStartSLOduration=2.720171771 podStartE2EDuration="2.720171771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:04.716733165 +0000 UTC m=+2862.768833119" watchObservedRunningTime="2026-10-30 18:00:04.720171771 +0000 UTC m=+2862.772271701" Oct 30 18:00:05.451475 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:05.451445 4965 generic.go:356] "Generic (PLEG): container finished" podID="4abe15c2-d8d1-40aa-a0d2-90673380e3ef" containerID="01cbc2400a46a536a49513cb378cfd59427af46632bf2aee158e64f5e1c4b646" exitCode=60 Oct 30 18:00:05.451976 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:05.451516 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb" event={"ID":"4abe15c2-d8d1-40aa-a0d2-90673380e3ef","Type":"ContainerDied","Data":"01cbc2400a46a536a49513cb378cfd59427af46632bf2aee158e64f5e1c4b646"} Oct 30 18:00:05.451976 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:05.451582 4965 scope.go:122] "RemoveContainer" containerID="066b41e0a642a0d06591fe4bbe672a42650e168513c51562d14e33626bafdd2b" Oct 30 18:00:05.451976 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:05.451834 4965 scope.go:122] "RemoveContainer" containerID="01cbc2400a46a536a49513cb378cfd59427af46632bf2aee158e64f5e1c4b646" Oct 30 18:00:05.452332 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:00:05.452297 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb_odh-ai-gateway-infra(4abe15c2-d8d1-40aa-a0d2-90673380e3ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb" podUID="4abe15c2-d8d1-40aa-a0d2-90673380e3ef" Oct 30 18:00:05.453685 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:05.453664 4965 generic.go:356] "Generic (PLEG): container finished" podID="02e21c00-730d-41ec-a53d-ea9f556c2459" containerID="d20474cdf0a35bcc396344e967bd5d2002b7aa318b9694c4a42fe37793ca545d" exitCode=60 Oct 30 18:00:05.453787 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:05.453763 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp" event={"ID":"02e21c00-730d-41ec-a53d-ea9f556c2459","Type":"ContainerDied","Data":"d20474cdf0a35bcc396344e967bd5d2002b7aa318b9694c4a42fe37793ca545d"} Oct 30 18:00:05.454080 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:05.454061 4965 scope.go:122] "RemoveContainer" containerID="d20474cdf0a35bcc396344e967bd5d2002b7aa318b9694c4a42fe37793ca545d" Oct 30 18:00:05.454414 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:00:05.454392 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp_odh-ai-gateway-infra(02e21c00-730d-41ec-a53d-ea9f556c2459)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp" podUID="02e21c00-730d-41ec-a53d-ea9f556c2459" Oct 30 18:00:05.455854 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:05.455835 4965 generic.go:356] "Generic (PLEG): container finished" podID="23dc886a-04cf-4d65-b4e5-b9de4c06e228" containerID="3a1c9bc836aadcc024c5a74616d0cd44ba3291e16cdb4fc48932f2ca7929c7d3" exitCode=60 Oct 30 18:00:05.455978 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:05.455905 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz" event={"ID":"23dc886a-04cf-4d65-b4e5-b9de4c06e228","Type":"ContainerDied","Data":"3a1c9bc836aadcc024c5a74616d0cd44ba3291e16cdb4fc48932f2ca7929c7d3"} Oct 30 18:00:05.456152 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:05.456133 4965 scope.go:122] "RemoveContainer" containerID="3a1c9bc836aadcc024c5a74616d0cd44ba3291e16cdb4fc48932f2ca7929c7d3" Oct 30 18:00:05.456459 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:00:05.456433 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz_odh-ai-gateway-infra(23dc886a-04cf-4d65-b4e5-b9de4c06e228)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz" podUID="23dc886a-04cf-4d65-b4e5-b9de4c06e228" Oct 30 18:00:05.457953 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:05.457933 4965 generic.go:356] "Generic (PLEG): container finished" podID="093a5e7e-2513-47f3-a742-cce2574ac917" containerID="ef1efc85355a3000742912381ae9a1723c8824b127781d81f989efbf43dc7156" exitCode=60 Oct 30 18:00:05.458059 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:05.457960 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88" event={"ID":"093a5e7e-2513-47f3-a742-cce2574ac917","Type":"ContainerDied","Data":"ef1efc85355a3000742912381ae9a1723c8824b127781d81f989efbf43dc7156"} Oct 30 18:00:05.458245 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:05.458230 4965 scope.go:122] "RemoveContainer" containerID="ef1efc85355a3000742912381ae9a1723c8824b127781d81f989efbf43dc7156" Oct 30 18:00:05.458526 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:00:05.458506 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88_odh-ai-gateway-infra(093a5e7e-2513-47f3-a742-cce2574ac917)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88" podUID="093a5e7e-2513-47f3-a742-cce2574ac917" Oct 30 18:00:05.459920 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:05.459899 4965 generic.go:356] "Generic (PLEG): container finished" podID="5e9c7d6c-3ded-495e-809b-5ffccf77b95a" containerID="c22612339c3c684db5c489e1ec9c33fbd113d367d673dd7e8fe00e3e49142923" exitCode=60 Oct 30 18:00:05.460027 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:05.459983 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw" event={"ID":"5e9c7d6c-3ded-495e-809b-5ffccf77b95a","Type":"ContainerDied","Data":"c22612339c3c684db5c489e1ec9c33fbd113d367d673dd7e8fe00e3e49142923"} Oct 30 18:00:05.460241 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:05.460227 4965 scope.go:122] "RemoveContainer" containerID="c22612339c3c684db5c489e1ec9c33fbd113d367d673dd7e8fe00e3e49142923" Oct 30 18:00:05.460520 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:00:05.460497 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw_odh-ai-gateway-infra(5e9c7d6c-3ded-495e-809b-5ffccf77b95a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw" podUID="5e9c7d6c-3ded-495e-809b-5ffccf77b95a" Oct 30 18:00:05.461736 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:05.461717 4965 generic.go:356] "Generic (PLEG): container finished" podID="63f5bed4-8de4-400c-a622-87e37d050d1c" containerID="f4a3f6f231ccf18d3802fc96319127b011a289ddafff99b956c0fc6b8bfd788a" exitCode=60 Oct 30 18:00:05.461830 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:05.461792 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj" event={"ID":"63f5bed4-8de4-400c-a622-87e37d050d1c","Type":"ContainerDied","Data":"f4a3f6f231ccf18d3802fc96319127b011a289ddafff99b956c0fc6b8bfd788a"} Oct 30 18:00:05.462052 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:05.462033 4965 scope.go:122] "RemoveContainer" containerID="f4a3f6f231ccf18d3802fc96319127b011a289ddafff99b956c0fc6b8bfd788a" Oct 30 18:00:05.462347 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:00:05.462327 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj_odh-ai-gateway-infra(63f5bed4-8de4-400c-a622-87e37d050d1c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj" podUID="63f5bed4-8de4-400c-a622-87e37d050d1c" Oct 30 18:00:05.463781 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:05.463762 4965 generic.go:356] "Generic (PLEG): container finished" podID="35173211-ac01-43fa-885c-b148a4f81ae4" containerID="a5fbb0b7c22e0cc229b1e28c15b9ab08f2d4fd158d0d153e2ad6ccc30b203c23" exitCode=60 Oct 30 18:00:05.463884 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:05.463828 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf" event={"ID":"35173211-ac01-43fa-885c-b148a4f81ae4","Type":"ContainerDied","Data":"a5fbb0b7c22e0cc229b1e28c15b9ab08f2d4fd158d0d153e2ad6ccc30b203c23"} Oct 30 18:00:05.464052 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:05.464034 4965 scope.go:122] "RemoveContainer" containerID="a5fbb0b7c22e0cc229b1e28c15b9ab08f2d4fd158d0d153e2ad6ccc30b203c23" Oct 30 18:00:05.464331 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:00:05.464309 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf_odh-ai-gateway-infra(35173211-ac01-43fa-885c-b148a4f81ae4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf" podUID="35173211-ac01-43fa-885c-b148a4f81ae4" Oct 30 18:00:05.466696 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:05.466681 4965 scope.go:122] "RemoveContainer" containerID="f6668b5e9077da189c3c24509de0b79358de39facee68199097f191bbb81b79d" Oct 30 18:00:05.481321 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:05.481298 4965 scope.go:122] "RemoveContainer" containerID="e2bc858f1b5003f65bdc9de24fc933b87867cc11139b7170a98a53d944b40549" Oct 30 18:00:05.494876 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:05.494848 4965 scope.go:122] "RemoveContainer" containerID="e6d1adff98b4ab927584fa2a5468b608dea63cfe9c62b44eca00fca87ca71615" Oct 30 18:00:05.519277 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:05.519253 4965 scope.go:122] "RemoveContainer" containerID="4aa281fb24788149960c27fbeb73cb1a88595bdb01025c56ca3b89df4cdc836b" Oct 30 18:00:05.532975 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:05.532952 4965 scope.go:122] "RemoveContainer" containerID="db91b63ad5bd643de73a3fca877f86969a348dfc990806f437b379d1f7490c39" Oct 30 18:00:05.547381 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:05.547359 4965 scope.go:122] "RemoveContainer" containerID="29639a11c34ee6847aec413cd7eb2a81cef7002528c0080ca04414b5a813df4e" Oct 30 18:00:05.998199 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:00:05.998164 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:00:06.473816 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:06.473785 4965 scope.go:122] "RemoveContainer" containerID="a5fbb0b7c22e0cc229b1e28c15b9ab08f2d4fd158d0d153e2ad6ccc30b203c23" Oct 30 18:00:06.474353 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:00:06.474214 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf_odh-ai-gateway-infra(35173211-ac01-43fa-885c-b148a4f81ae4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf" podUID="35173211-ac01-43fa-885c-b148a4f81ae4" Oct 30 18:00:06.475450 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:06.475427 4965 scope.go:122] "RemoveContainer" containerID="01cbc2400a46a536a49513cb378cfd59427af46632bf2aee158e64f5e1c4b646" Oct 30 18:00:06.475769 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:00:06.475741 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb_odh-ai-gateway-infra(4abe15c2-d8d1-40aa-a0d2-90673380e3ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb" podUID="4abe15c2-d8d1-40aa-a0d2-90673380e3ef" Oct 30 18:00:06.477146 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:06.477125 4965 scope.go:122] "RemoveContainer" containerID="d20474cdf0a35bcc396344e967bd5d2002b7aa318b9694c4a42fe37793ca545d" Oct 30 18:00:06.477474 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:00:06.477452 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp_odh-ai-gateway-infra(02e21c00-730d-41ec-a53d-ea9f556c2459)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp" podUID="02e21c00-730d-41ec-a53d-ea9f556c2459" Oct 30 18:00:06.478973 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:06.478946 4965 scope.go:122] "RemoveContainer" containerID="3a1c9bc836aadcc024c5a74616d0cd44ba3291e16cdb4fc48932f2ca7929c7d3" Oct 30 18:00:06.479286 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:00:06.479258 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz_odh-ai-gateway-infra(23dc886a-04cf-4d65-b4e5-b9de4c06e228)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz" podUID="23dc886a-04cf-4d65-b4e5-b9de4c06e228" Oct 30 18:00:06.480467 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:06.480444 4965 scope.go:122] "RemoveContainer" containerID="ef1efc85355a3000742912381ae9a1723c8824b127781d81f989efbf43dc7156" Oct 30 18:00:06.480734 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:00:06.480713 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88_odh-ai-gateway-infra(093a5e7e-2513-47f3-a742-cce2574ac917)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88" podUID="093a5e7e-2513-47f3-a742-cce2574ac917" Oct 30 18:00:06.481954 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:06.481935 4965 scope.go:122] "RemoveContainer" containerID="c22612339c3c684db5c489e1ec9c33fbd113d367d673dd7e8fe00e3e49142923" Oct 30 18:00:06.482199 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:00:06.482178 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw_odh-ai-gateway-infra(5e9c7d6c-3ded-495e-809b-5ffccf77b95a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw" podUID="5e9c7d6c-3ded-495e-809b-5ffccf77b95a" Oct 30 18:00:06.483521 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:06.483504 4965 scope.go:122] "RemoveContainer" containerID="f4a3f6f231ccf18d3802fc96319127b011a289ddafff99b956c0fc6b8bfd788a" Oct 30 18:00:06.483718 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:00:06.483700 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj_odh-ai-gateway-infra(63f5bed4-8de4-400c-a622-87e37d050d1c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj" podUID="63f5bed4-8de4-400c-a622-87e37d050d1c" Oct 30 18:00:06.820833 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:06.820753 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-785eb5bf-tgr2l"] Oct 30 18:00:06.821607 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:06.821590 4965 cpu_manager.go:414] "containerMap: removing container" podUID="a2f007f6-27a0-4cfb-ba7e-c20f030397f1" containerName="main" Oct 30 18:00:06.821648 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:06.821609 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2f007f6-27a0-4cfb-ba7e-c20f030397f1" containerName="main" Oct 30 18:00:06.821786 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:06.821777 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2f007f6-27a0-4cfb-ba7e-c20f030397f1" containerName="main" Oct 30 18:00:06.826928 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:06.826903 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785eb5bf-tgr2l" Oct 30 18:00:06.841398 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:06.841370 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:00:06.843021 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:06.842998 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-785eb5bf-tgr2l"] Oct 30 18:00:06.921225 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:06.921185 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ab0275-09f7-45c9-bc2b-4e9893421b38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-785eb5bf-tgr2l\" (UID: \"43ab0275-09f7-45c9-bc2b-4e9893421b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785eb5bf-tgr2l" Oct 30 18:00:06.921382 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:06.921351 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n44dh\" (UniqueName: \"kubernetes.io/projected/43ab0275-09f7-45c9-bc2b-4e9893421b38-kube-api-access-n44dh\") pod \"maas-api-revoke-sub-785eb5bf-tgr2l\" (UID: \"43ab0275-09f7-45c9-bc2b-4e9893421b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785eb5bf-tgr2l" Oct 30 18:00:06.921509 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:06.921494 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ab0275-09f7-45c9-bc2b-4e9893421b38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-785eb5bf-tgr2l\" (UID: \"43ab0275-09f7-45c9-bc2b-4e9893421b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785eb5bf-tgr2l" Oct 30 18:00:07.023344 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:07.023307 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ab0275-09f7-45c9-bc2b-4e9893421b38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-785eb5bf-tgr2l\" (UID: \"43ab0275-09f7-45c9-bc2b-4e9893421b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785eb5bf-tgr2l" Oct 30 18:00:07.023527 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:07.023404 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n44dh\" (UniqueName: \"kubernetes.io/projected/43ab0275-09f7-45c9-bc2b-4e9893421b38-kube-api-access-n44dh\") pod \"maas-api-revoke-sub-785eb5bf-tgr2l\" (UID: \"43ab0275-09f7-45c9-bc2b-4e9893421b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785eb5bf-tgr2l" Oct 30 18:00:07.023527 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:07.023493 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ab0275-09f7-45c9-bc2b-4e9893421b38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-785eb5bf-tgr2l\" (UID: \"43ab0275-09f7-45c9-bc2b-4e9893421b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785eb5bf-tgr2l" Oct 30 18:00:07.024057 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:07.024032 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ab0275-09f7-45c9-bc2b-4e9893421b38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-785eb5bf-tgr2l\" (UID: \"43ab0275-09f7-45c9-bc2b-4e9893421b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785eb5bf-tgr2l" Oct 30 18:00:07.025901 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:07.025884 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ab0275-09f7-45c9-bc2b-4e9893421b38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-785eb5bf-tgr2l\" (UID: \"43ab0275-09f7-45c9-bc2b-4e9893421b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785eb5bf-tgr2l" Oct 30 18:00:07.048795 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:07.048768 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n44dh\" (UniqueName: \"kubernetes.io/projected/43ab0275-09f7-45c9-bc2b-4e9893421b38-kube-api-access-n44dh\") pod \"maas-api-revoke-sub-785eb5bf-tgr2l\" (UID: \"43ab0275-09f7-45c9-bc2b-4e9893421b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785eb5bf-tgr2l" Oct 30 18:00:07.146158 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:07.146128 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785eb5bf-tgr2l" Oct 30 18:00:07.453186 ip-10-0-139-222 kubenswrapper[4965]: W1030 18:00:07.453148 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ab0275_09f7_45c9_bc2b_4e9893421b38.slice/crio-b0de4343e0f6c2a9bf016c000799aef7f45f962b2730591b2ca77f4ab855d9d1 WatchSource:0}: Error finding container b0de4343e0f6c2a9bf016c000799aef7f45f962b2730591b2ca77f4ab855d9d1: Status 404 returned error can't find the container with id b0de4343e0f6c2a9bf016c000799aef7f45f962b2730591b2ca77f4ab855d9d1 Oct 30 18:00:07.454583 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:07.454554 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-785eb5bf-tgr2l"] Oct 30 18:00:07.494478 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:07.494449 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785eb5bf-tgr2l" event={"ID":"43ab0275-09f7-45c9-bc2b-4e9893421b38","Type":"ContainerStarted","Data":"b0de4343e0f6c2a9bf016c000799aef7f45f962b2730591b2ca77f4ab855d9d1"} Oct 30 18:00:08.503584 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:08.503545 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785eb5bf-tgr2l" event={"ID":"43ab0275-09f7-45c9-bc2b-4e9893421b38","Type":"ContainerStarted","Data":"df7e7f18a0b17a2c3bc90a595d12875a5534f31d2350033cf1effc445ef01eae"} Oct 30 18:00:08.545224 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:08.545166 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785eb5bf-tgr2l" podStartSLOduration=2.545152406 podStartE2EDuration="2.545152406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:08.542488243 +0000 UTC m=+2866.594588174" watchObservedRunningTime="2026-10-30 18:00:08.545152406 +0000 UTC m=+2866.597252337" Oct 30 18:00:09.514099 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:09.514063 4965 generic.go:356] "Generic (PLEG): container finished" podID="43ab0275-09f7-45c9-bc2b-4e9893421b38" containerID="df7e7f18a0b17a2c3bc90a595d12875a5534f31d2350033cf1effc445ef01eae" exitCode=0 Oct 30 18:00:09.514537 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:09.514154 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785eb5bf-tgr2l" event={"ID":"43ab0275-09f7-45c9-bc2b-4e9893421b38","Type":"ContainerDied","Data":"df7e7f18a0b17a2c3bc90a595d12875a5534f31d2350033cf1effc445ef01eae"} Oct 30 18:00:10.753938 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:10.752516 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785eb5bf-tgr2l" Oct 30 18:00:10.895297 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:10.895264 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43ab0275-09f7-45c9-bc2b-4e9893421b38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ab0275-09f7-45c9-bc2b-4e9893421b38-maas-api-openshift-ca\") pod \"43ab0275-09f7-45c9-bc2b-4e9893421b38\" (UID: \"43ab0275-09f7-45c9-bc2b-4e9893421b38\") " Oct 30 18:00:10.895469 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:10.895421 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43ab0275-09f7-45c9-bc2b-4e9893421b38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ab0275-09f7-45c9-bc2b-4e9893421b38-maas-api-xks-ca\") pod \"43ab0275-09f7-45c9-bc2b-4e9893421b38\" (UID: \"43ab0275-09f7-45c9-bc2b-4e9893421b38\") " Oct 30 18:00:10.895572 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:10.895556 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43ab0275-09f7-45c9-bc2b-4e9893421b38-kube-api-access-n44dh\" (UniqueName: \"kubernetes.io/projected/43ab0275-09f7-45c9-bc2b-4e9893421b38-kube-api-access-n44dh\") pod \"43ab0275-09f7-45c9-bc2b-4e9893421b38\" (UID: \"43ab0275-09f7-45c9-bc2b-4e9893421b38\") " Oct 30 18:00:10.895635 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:10.895608 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ab0275-09f7-45c9-bc2b-4e9893421b38-maas-api-openshift-ca" pod "43ab0275-09f7-45c9-bc2b-4e9893421b38" (UID: "43ab0275-09f7-45c9-bc2b-4e9893421b38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:00:10.896173 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:10.896149 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ab0275-09f7-45c9-bc2b-4e9893421b38-maas-api-openshift-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:00:10.897629 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:10.897583 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ab0275-09f7-45c9-bc2b-4e9893421b38-kube-api-access-n44dh" pod "43ab0275-09f7-45c9-bc2b-4e9893421b38" (UID: "43ab0275-09f7-45c9-bc2b-4e9893421b38"). InnerVolumeSpecName "kube-api-access-n44dh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:00:10.897707 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:10.897646 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ab0275-09f7-45c9-bc2b-4e9893421b38-maas-api-xks-ca" pod "43ab0275-09f7-45c9-bc2b-4e9893421b38" (UID: "43ab0275-09f7-45c9-bc2b-4e9893421b38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:00:10.997659 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:10.997625 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n44dh\" (UniqueName: \"kubernetes.io/projected/43ab0275-09f7-45c9-bc2b-4e9893421b38-kube-api-access-n44dh\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:00:10.997659 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:10.997657 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ab0275-09f7-45c9-bc2b-4e9893421b38-maas-api-xks-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:00:11.531194 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:11.531164 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785eb5bf-tgr2l" Oct 30 18:00:11.531357 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:11.531201 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785eb5bf-tgr2l" event={"ID":"43ab0275-09f7-45c9-bc2b-4e9893421b38","Type":"ContainerDied","Data":"b0de4343e0f6c2a9bf016c000799aef7f45f962b2730591b2ca77f4ab855d9d1"} Oct 30 18:00:11.531357 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:11.531249 4965 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0de4343e0f6c2a9bf016c000799aef7f45f962b2730591b2ca77f4ab855d9d1" Oct 30 18:00:15.091595 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:15.091561 4965 scope.go:122] "RemoveContainer" containerID="01cbc2400a46a536a49513cb378cfd59427af46632bf2aee158e64f5e1c4b646" Oct 30 18:00:15.092057 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:15.091739 4965 scope.go:122] "RemoveContainer" containerID="d20474cdf0a35bcc396344e967bd5d2002b7aa318b9694c4a42fe37793ca545d" Oct 30 18:00:15.092125 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:15.092083 4965 scope.go:122] "RemoveContainer" containerID="3a1c9bc836aadcc024c5a74616d0cd44ba3291e16cdb4fc48932f2ca7929c7d3" Oct 30 18:00:15.466413 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:15.466385 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-6d96cfd86b-kmrsv" Oct 30 18:00:15.607016 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:15.606982 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp" event={"ID":"02e21c00-730d-41ec-a53d-ea9f556c2459","Type":"ContainerStarted","Data":"96ea1b1ae96d94610376db1b4d42f77e6e370f60247798090a259a41684e26d8"} Oct 30 18:00:16.099140 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:16.097162 4965 scope.go:122] "RemoveContainer" containerID="ef1efc85355a3000742912381ae9a1723c8824b127781d81f989efbf43dc7156" Oct 30 18:00:16.617816 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:16.617748 4965 generic.go:356] "Generic (PLEG): container finished" podID="02e21c00-730d-41ec-a53d-ea9f556c2459" containerID="96ea1b1ae96d94610376db1b4d42f77e6e370f60247798090a259a41684e26d8" exitCode=60 Oct 30 18:00:16.617955 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:16.617817 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp" event={"ID":"02e21c00-730d-41ec-a53d-ea9f556c2459","Type":"ContainerDied","Data":"96ea1b1ae96d94610376db1b4d42f77e6e370f60247798090a259a41684e26d8"} Oct 30 18:00:16.617955 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:16.617884 4965 scope.go:122] "RemoveContainer" containerID="d20474cdf0a35bcc396344e967bd5d2002b7aa318b9694c4a42fe37793ca545d" Oct 30 18:00:16.618370 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:16.618347 4965 scope.go:122] "RemoveContainer" containerID="96ea1b1ae96d94610376db1b4d42f77e6e370f60247798090a259a41684e26d8" Oct 30 18:00:16.618816 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:00:16.618747 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp_odh-ai-gateway-infra(02e21c00-730d-41ec-a53d-ea9f556c2459)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp" podUID="02e21c00-730d-41ec-a53d-ea9f556c2459" Oct 30 18:00:16.620152 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:16.620134 4965 generic.go:356] "Generic (PLEG): container finished" podID="23dc886a-04cf-4d65-b4e5-b9de4c06e228" containerID="fcaae44b7fb3df6338b3669395e3568cf20248e5b37c882c5f27c6205d651bba" exitCode=60 Oct 30 18:00:16.620254 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:16.620184 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz" event={"ID":"23dc886a-04cf-4d65-b4e5-b9de4c06e228","Type":"ContainerDied","Data":"fcaae44b7fb3df6338b3669395e3568cf20248e5b37c882c5f27c6205d651bba"} Oct 30 18:00:16.620585 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:16.620565 4965 scope.go:122] "RemoveContainer" containerID="fcaae44b7fb3df6338b3669395e3568cf20248e5b37c882c5f27c6205d651bba" Oct 30 18:00:16.620931 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:00:16.620912 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz_odh-ai-gateway-infra(23dc886a-04cf-4d65-b4e5-b9de4c06e228)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz" podUID="23dc886a-04cf-4d65-b4e5-b9de4c06e228" Oct 30 18:00:16.622152 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:16.622131 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88" event={"ID":"093a5e7e-2513-47f3-a742-cce2574ac917","Type":"ContainerStarted","Data":"5e29500b1b48c54f93e157fb42c9ba5004812476290340304dac857f0a64782e"} Oct 30 18:00:16.624057 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:16.624037 4965 generic.go:356] "Generic (PLEG): container finished" podID="4abe15c2-d8d1-40aa-a0d2-90673380e3ef" containerID="73dcb020644ef40bfe9a93c79a24353c3780d0d3569b84c096746bd95a5be276" exitCode=60 Oct 30 18:00:16.624177 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:16.624067 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb" event={"ID":"4abe15c2-d8d1-40aa-a0d2-90673380e3ef","Type":"ContainerDied","Data":"73dcb020644ef40bfe9a93c79a24353c3780d0d3569b84c096746bd95a5be276"} Oct 30 18:00:16.624381 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:16.624365 4965 scope.go:122] "RemoveContainer" containerID="73dcb020644ef40bfe9a93c79a24353c3780d0d3569b84c096746bd95a5be276" Oct 30 18:00:16.624606 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:00:16.624586 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb_odh-ai-gateway-infra(4abe15c2-d8d1-40aa-a0d2-90673380e3ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb" podUID="4abe15c2-d8d1-40aa-a0d2-90673380e3ef" Oct 30 18:00:16.631790 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:16.631773 4965 scope.go:122] "RemoveContainer" containerID="3a1c9bc836aadcc024c5a74616d0cd44ba3291e16cdb4fc48932f2ca7929c7d3" Oct 30 18:00:16.646611 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:16.646591 4965 scope.go:122] "RemoveContainer" containerID="01cbc2400a46a536a49513cb378cfd59427af46632bf2aee158e64f5e1c4b646" Oct 30 18:00:16.670044 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:16.670018 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp"] Oct 30 18:00:17.091272 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.091247 4965 scope.go:122] "RemoveContainer" containerID="f4a3f6f231ccf18d3802fc96319127b011a289ddafff99b956c0fc6b8bfd788a" Oct 30 18:00:17.465482 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.465449 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b52a3f74-l9swh"] Oct 30 18:00:17.466425 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.466402 4965 cpu_manager.go:414] "containerMap: removing container" podUID="43ab0275-09f7-45c9-bc2b-4e9893421b38" containerName="revoke-keys" Oct 30 18:00:17.466425 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.466423 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ab0275-09f7-45c9-bc2b-4e9893421b38" containerName="revoke-keys" Oct 30 18:00:17.466585 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.466571 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ab0275-09f7-45c9-bc2b-4e9893421b38" containerName="revoke-keys" Oct 30 18:00:17.470493 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.470471 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52a3f74-l9swh" Oct 30 18:00:17.485539 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.485504 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:00:17.489136 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.489113 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b52a3f74-l9swh"] Oct 30 18:00:17.548949 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.548914 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-584a12-6869fbf48b-jwbm4"] Oct 30 18:00:17.553964 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.553940 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-6869fbf48b-jwbm4" Oct 30 18:00:17.554419 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.554398 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxkj9\" (UniqueName: \"kubernetes.io/projected/5e43b107-9066-4368-a04a-8437c7b4201d-kube-api-access-vxkj9\") pod \"maas-api-revoke-sub-b52a3f74-l9swh\" (UID: \"5e43b107-9066-4368-a04a-8437c7b4201d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52a3f74-l9swh" Oct 30 18:00:17.554580 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.554560 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e43b107-9066-4368-a04a-8437c7b4201d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b52a3f74-l9swh\" (UID: \"5e43b107-9066-4368-a04a-8437c7b4201d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52a3f74-l9swh" Oct 30 18:00:17.554779 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.554763 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e43b107-9066-4368-a04a-8437c7b4201d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b52a3f74-l9swh\" (UID: \"5e43b107-9066-4368-a04a-8437c7b4201d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52a3f74-l9swh" Oct 30 18:00:17.570966 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.570947 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-584a12\"" Oct 30 18:00:17.573800 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.573774 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-584a12-6869fbf48b-jwbm4"] Oct 30 18:00:17.641486 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.641453 4965 generic.go:356] "Generic (PLEG): container finished" podID="093a5e7e-2513-47f3-a742-cce2574ac917" containerID="5e29500b1b48c54f93e157fb42c9ba5004812476290340304dac857f0a64782e" exitCode=60 Oct 30 18:00:17.641649 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.641531 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88" event={"ID":"093a5e7e-2513-47f3-a742-cce2574ac917","Type":"ContainerDied","Data":"5e29500b1b48c54f93e157fb42c9ba5004812476290340304dac857f0a64782e"} Oct 30 18:00:17.641649 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.641606 4965 scope.go:122] "RemoveContainer" containerID="ef1efc85355a3000742912381ae9a1723c8824b127781d81f989efbf43dc7156" Oct 30 18:00:17.642104 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.642081 4965 scope.go:122] "RemoveContainer" containerID="5e29500b1b48c54f93e157fb42c9ba5004812476290340304dac857f0a64782e" Oct 30 18:00:17.642532 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:00:17.642510 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88_odh-ai-gateway-infra(093a5e7e-2513-47f3-a742-cce2574ac917)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88" podUID="093a5e7e-2513-47f3-a742-cce2574ac917" Oct 30 18:00:17.646403 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.646380 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj" event={"ID":"63f5bed4-8de4-400c-a622-87e37d050d1c","Type":"ContainerStarted","Data":"9bdbd69e9dc32868f8c12f4f09110e26042d2c9fff7174de168a3d50c07b9e41"} Oct 30 18:00:17.655901 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.655854 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e43b107-9066-4368-a04a-8437c7b4201d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b52a3f74-l9swh\" (UID: \"5e43b107-9066-4368-a04a-8437c7b4201d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52a3f74-l9swh" Oct 30 18:00:17.656367 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.656034 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c18e28aa-2d36-480c-9284-4b3b7212539e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-584a12-6869fbf48b-jwbm4\" (UID: \"c18e28aa-2d36-480c-9284-4b3b7212539e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-6869fbf48b-jwbm4" Oct 30 18:00:17.656367 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.656091 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxnn4\" (UniqueName: \"kubernetes.io/projected/c18e28aa-2d36-480c-9284-4b3b7212539e-kube-api-access-rxnn4\") pod \"payload-processing-e2e-ipp-a-584a12-6869fbf48b-jwbm4\" (UID: \"c18e28aa-2d36-480c-9284-4b3b7212539e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-6869fbf48b-jwbm4" Oct 30 18:00:17.656562 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.656431 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e43b107-9066-4368-a04a-8437c7b4201d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b52a3f74-l9swh\" (UID: \"5e43b107-9066-4368-a04a-8437c7b4201d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52a3f74-l9swh" Oct 30 18:00:17.657072 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.657039 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e43b107-9066-4368-a04a-8437c7b4201d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b52a3f74-l9swh\" (UID: \"5e43b107-9066-4368-a04a-8437c7b4201d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52a3f74-l9swh" Oct 30 18:00:17.657202 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.657165 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxkj9\" (UniqueName: \"kubernetes.io/projected/5e43b107-9066-4368-a04a-8437c7b4201d-kube-api-access-vxkj9\") pod \"maas-api-revoke-sub-b52a3f74-l9swh\" (UID: \"5e43b107-9066-4368-a04a-8437c7b4201d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52a3f74-l9swh" Oct 30 18:00:17.669719 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.669693 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e43b107-9066-4368-a04a-8437c7b4201d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b52a3f74-l9swh\" (UID: \"5e43b107-9066-4368-a04a-8437c7b4201d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52a3f74-l9swh" Oct 30 18:00:17.900925 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.900258 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c18e28aa-2d36-480c-9284-4b3b7212539e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-584a12-6869fbf48b-jwbm4\" (UID: \"c18e28aa-2d36-480c-9284-4b3b7212539e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-6869fbf48b-jwbm4" Oct 30 18:00:17.900925 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.900330 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxnn4\" (UniqueName: \"kubernetes.io/projected/c18e28aa-2d36-480c-9284-4b3b7212539e-kube-api-access-rxnn4\") pod \"payload-processing-e2e-ipp-a-584a12-6869fbf48b-jwbm4\" (UID: \"c18e28aa-2d36-480c-9284-4b3b7212539e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-6869fbf48b-jwbm4" Oct 30 18:00:17.906063 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.904140 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c18e28aa-2d36-480c-9284-4b3b7212539e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-584a12-6869fbf48b-jwbm4\" (UID: \"c18e28aa-2d36-480c-9284-4b3b7212539e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-6869fbf48b-jwbm4" Oct 30 18:00:17.906063 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.904721 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88"] Oct 30 18:00:17.932990 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.932786 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxnn4\" (UniqueName: \"kubernetes.io/projected/c18e28aa-2d36-480c-9284-4b3b7212539e-kube-api-access-rxnn4\") pod \"payload-processing-e2e-ipp-a-584a12-6869fbf48b-jwbm4\" (UID: \"c18e28aa-2d36-480c-9284-4b3b7212539e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-6869fbf48b-jwbm4" Oct 30 18:00:17.948440 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.948405 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxkj9\" (UniqueName: \"kubernetes.io/projected/5e43b107-9066-4368-a04a-8437c7b4201d-kube-api-access-vxkj9\") pod \"maas-api-revoke-sub-b52a3f74-l9swh\" (UID: \"5e43b107-9066-4368-a04a-8437c7b4201d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52a3f74-l9swh" Oct 30 18:00:17.952064 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.952038 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb"] Oct 30 18:00:17.963324 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:17.963301 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz"] Oct 30 18:00:18.090813 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.090785 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52a3f74-l9swh" Oct 30 18:00:18.092057 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.091972 4965 scope.go:122] "RemoveContainer" containerID="a5fbb0b7c22e0cc229b1e28c15b9ab08f2d4fd158d0d153e2ad6ccc30b203c23" Oct 30 18:00:18.092931 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.092496 4965 scope.go:122] "RemoveContainer" containerID="c22612339c3c684db5c489e1ec9c33fbd113d367d673dd7e8fe00e3e49142923" Oct 30 18:00:18.113346 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.113321 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp" Oct 30 18:00:18.171118 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.171058 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-6869fbf48b-jwbm4" Oct 30 18:00:18.202117 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.202086 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb" Oct 30 18:00:18.218179 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.210506 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02e21c00-730d-41ec-a53d-ea9f556c2459-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02e21c00-730d-41ec-a53d-ea9f556c2459-maas-api-service-ca\") pod \"02e21c00-730d-41ec-a53d-ea9f556c2459\" (UID: \"02e21c00-730d-41ec-a53d-ea9f556c2459\") " Oct 30 18:00:18.218882 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.218813 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02e21c00-730d-41ec-a53d-ea9f556c2459-kube-api-access-nr8zx\" (UniqueName: \"kubernetes.io/projected/02e21c00-730d-41ec-a53d-ea9f556c2459-kube-api-access-nr8zx\") pod \"02e21c00-730d-41ec-a53d-ea9f556c2459\" (UID: \"02e21c00-730d-41ec-a53d-ea9f556c2459\") " Oct 30 18:00:18.221639 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.221596 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02e21c00-730d-41ec-a53d-ea9f556c2459-maas-api-service-ca" pod "02e21c00-730d-41ec-a53d-ea9f556c2459" (UID: "02e21c00-730d-41ec-a53d-ea9f556c2459"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:00:18.226187 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.226147 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02e21c00-730d-41ec-a53d-ea9f556c2459-kube-api-access-nr8zx" pod "02e21c00-730d-41ec-a53d-ea9f556c2459" (UID: "02e21c00-730d-41ec-a53d-ea9f556c2459"). InnerVolumeSpecName "kube-api-access-nr8zx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:00:18.252228 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.252196 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz" Oct 30 18:00:18.322186 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.321181 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4abe15c2-d8d1-40aa-a0d2-90673380e3ef-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4abe15c2-d8d1-40aa-a0d2-90673380e3ef-maas-api-service-ca\") pod \"4abe15c2-d8d1-40aa-a0d2-90673380e3ef\" (UID: \"4abe15c2-d8d1-40aa-a0d2-90673380e3ef\") " Oct 30 18:00:18.322186 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.321458 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4abe15c2-d8d1-40aa-a0d2-90673380e3ef-kube-api-access-7898p\" (UniqueName: \"kubernetes.io/projected/4abe15c2-d8d1-40aa-a0d2-90673380e3ef-kube-api-access-7898p\") pod \"4abe15c2-d8d1-40aa-a0d2-90673380e3ef\" (UID: \"4abe15c2-d8d1-40aa-a0d2-90673380e3ef\") " Oct 30 18:00:18.322186 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.321820 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4abe15c2-d8d1-40aa-a0d2-90673380e3ef-maas-api-service-ca" pod "4abe15c2-d8d1-40aa-a0d2-90673380e3ef" (UID: "4abe15c2-d8d1-40aa-a0d2-90673380e3ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:00:18.322502 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.322283 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4abe15c2-d8d1-40aa-a0d2-90673380e3ef-maas-api-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:00:18.322502 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.322308 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr8zx\" (UniqueName: \"kubernetes.io/projected/02e21c00-730d-41ec-a53d-ea9f556c2459-kube-api-access-nr8zx\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:00:18.322502 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.322323 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02e21c00-730d-41ec-a53d-ea9f556c2459-maas-api-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:00:18.375308 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.375176 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4abe15c2-d8d1-40aa-a0d2-90673380e3ef-kube-api-access-7898p" pod "4abe15c2-d8d1-40aa-a0d2-90673380e3ef" (UID: "4abe15c2-d8d1-40aa-a0d2-90673380e3ef"). InnerVolumeSpecName "kube-api-access-7898p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:00:18.424151 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.424057 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23dc886a-04cf-4d65-b4e5-b9de4c06e228-kube-api-access-8jgqc\" (UniqueName: \"kubernetes.io/projected/23dc886a-04cf-4d65-b4e5-b9de4c06e228-kube-api-access-8jgqc\") pod \"23dc886a-04cf-4d65-b4e5-b9de4c06e228\" (UID: \"23dc886a-04cf-4d65-b4e5-b9de4c06e228\") " Oct 30 18:00:18.424151 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.424130 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23dc886a-04cf-4d65-b4e5-b9de4c06e228-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23dc886a-04cf-4d65-b4e5-b9de4c06e228-maas-api-service-ca\") pod \"23dc886a-04cf-4d65-b4e5-b9de4c06e228\" (UID: \"23dc886a-04cf-4d65-b4e5-b9de4c06e228\") " Oct 30 18:00:18.425163 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.424911 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7898p\" (UniqueName: \"kubernetes.io/projected/4abe15c2-d8d1-40aa-a0d2-90673380e3ef-kube-api-access-7898p\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:00:18.425898 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.425455 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23dc886a-04cf-4d65-b4e5-b9de4c06e228-maas-api-service-ca" pod "23dc886a-04cf-4d65-b4e5-b9de4c06e228" (UID: "23dc886a-04cf-4d65-b4e5-b9de4c06e228"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:00:18.446658 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.446609 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23dc886a-04cf-4d65-b4e5-b9de4c06e228-kube-api-access-8jgqc" pod "23dc886a-04cf-4d65-b4e5-b9de4c06e228" (UID: "23dc886a-04cf-4d65-b4e5-b9de4c06e228"). InnerVolumeSpecName "kube-api-access-8jgqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:00:18.487621 ip-10-0-139-222 kubenswrapper[4965]: W1030 18:00:18.487586 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e43b107_9066_4368_a04a_8437c7b4201d.slice/crio-94f85682dc54250b94ad91fe931f00704ada991b384d53be4e6bc98045c4ec62 WatchSource:0}: Error finding container 94f85682dc54250b94ad91fe931f00704ada991b384d53be4e6bc98045c4ec62: Status 404 returned error can't find the container with id 94f85682dc54250b94ad91fe931f00704ada991b384d53be4e6bc98045c4ec62 Oct 30 18:00:18.490648 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.490620 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b52a3f74-l9swh"] Oct 30 18:00:18.526492 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.526260 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jgqc\" (UniqueName: \"kubernetes.io/projected/23dc886a-04cf-4d65-b4e5-b9de4c06e228-kube-api-access-8jgqc\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:00:18.526492 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.526289 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23dc886a-04cf-4d65-b4e5-b9de4c06e228-maas-api-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:00:18.530961 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.530938 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-584a12-6869fbf48b-jwbm4"] Oct 30 18:00:18.537244 ip-10-0-139-222 kubenswrapper[4965]: W1030 18:00:18.537215 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18e28aa_2d36_480c_9284_4b3b7212539e.slice/crio-d6c6c5eefd625d4a632edf24576a1a20562b08150cc46242b62de320552c59d3 WatchSource:0}: Error finding container d6c6c5eefd625d4a632edf24576a1a20562b08150cc46242b62de320552c59d3: Status 404 returned error can't find the container with id d6c6c5eefd625d4a632edf24576a1a20562b08150cc46242b62de320552c59d3 Oct 30 18:00:18.657807 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.657769 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz" Oct 30 18:00:18.657988 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.657817 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz" event={"ID":"23dc886a-04cf-4d65-b4e5-b9de4c06e228","Type":"ContainerDied","Data":"002c3d72e7ad42a49c76582180605559e51bd4e33ec2593031c9df44d69e0b06"} Oct 30 18:00:18.657988 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.657930 4965 scope.go:122] "RemoveContainer" containerID="fcaae44b7fb3df6338b3669395e3568cf20248e5b37c882c5f27c6205d651bba" Oct 30 18:00:18.660847 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.660412 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52a3f74-l9swh" event={"ID":"5e43b107-9066-4368-a04a-8437c7b4201d","Type":"ContainerStarted","Data":"94f85682dc54250b94ad91fe931f00704ada991b384d53be4e6bc98045c4ec62"} Oct 30 18:00:18.662210 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.662182 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-6869fbf48b-jwbm4" event={"ID":"c18e28aa-2d36-480c-9284-4b3b7212539e","Type":"ContainerStarted","Data":"d6c6c5eefd625d4a632edf24576a1a20562b08150cc46242b62de320552c59d3"} Oct 30 18:00:18.669092 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.668333 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw" event={"ID":"5e9c7d6c-3ded-495e-809b-5ffccf77b95a","Type":"ContainerStarted","Data":"7af310214cb7a0d685c7d0cb9e15d7b114148efa43bf147e557e0f1613cc0a9f"} Oct 30 18:00:18.671620 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.671598 4965 generic.go:356] "Generic (PLEG): container finished" podID="63f5bed4-8de4-400c-a622-87e37d050d1c" containerID="9bdbd69e9dc32868f8c12f4f09110e26042d2c9fff7174de168a3d50c07b9e41" exitCode=60 Oct 30 18:00:18.673076 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.671770 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj" event={"ID":"63f5bed4-8de4-400c-a622-87e37d050d1c","Type":"ContainerDied","Data":"9bdbd69e9dc32868f8c12f4f09110e26042d2c9fff7174de168a3d50c07b9e41"} Oct 30 18:00:18.673076 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.672331 4965 scope.go:122] "RemoveContainer" containerID="9bdbd69e9dc32868f8c12f4f09110e26042d2c9fff7174de168a3d50c07b9e41" Oct 30 18:00:18.673076 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:00:18.672706 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj_odh-ai-gateway-infra(63f5bed4-8de4-400c-a622-87e37d050d1c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj" podUID="63f5bed4-8de4-400c-a622-87e37d050d1c" Oct 30 18:00:18.675258 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.675206 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb" event={"ID":"4abe15c2-d8d1-40aa-a0d2-90673380e3ef","Type":"ContainerDied","Data":"b7b8c6bc62c7c5e43e633db0517192c8bab1358bd556a4f7de1467b40df37569"} Oct 30 18:00:18.675335 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.675295 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb" Oct 30 18:00:18.678740 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.678714 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp" event={"ID":"02e21c00-730d-41ec-a53d-ea9f556c2459","Type":"ContainerDied","Data":"8885466fc7eeed3f68cb1ac364357c941591ea4ef8af01f1cd437e619c78c6e9"} Oct 30 18:00:18.678816 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.678803 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp" Oct 30 18:00:18.724678 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.724650 4965 scope.go:122] "RemoveContainer" containerID="f4a3f6f231ccf18d3802fc96319127b011a289ddafff99b956c0fc6b8bfd788a" Oct 30 18:00:18.763433 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.753249 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj"] Oct 30 18:00:18.765911 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.765888 4965 scope.go:122] "RemoveContainer" containerID="73dcb020644ef40bfe9a93c79a24353c3780d0d3569b84c096746bd95a5be276" Oct 30 18:00:18.828566 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.828447 4965 scope.go:122] "RemoveContainer" containerID="96ea1b1ae96d94610376db1b4d42f77e6e370f60247798090a259a41684e26d8" Oct 30 18:00:18.877295 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.877275 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88" Oct 30 18:00:18.888257 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.888230 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb"] Oct 30 18:00:18.893546 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.893520 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-90b48a-29889720-bpflb"] Oct 30 18:00:18.927010 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.926929 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz"] Oct 30 18:00:18.930797 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.930765 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/093a5e7e-2513-47f3-a742-cce2574ac917-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/093a5e7e-2513-47f3-a742-cce2574ac917-maas-api-service-ca\") pod \"093a5e7e-2513-47f3-a742-cce2574ac917\" (UID: \"093a5e7e-2513-47f3-a742-cce2574ac917\") " Oct 30 18:00:18.930954 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.930935 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/093a5e7e-2513-47f3-a742-cce2574ac917-kube-api-access-6zv75\" (UniqueName: \"kubernetes.io/projected/093a5e7e-2513-47f3-a742-cce2574ac917-kube-api-access-6zv75\") pod \"093a5e7e-2513-47f3-a742-cce2574ac917\" (UID: \"093a5e7e-2513-47f3-a742-cce2574ac917\") " Oct 30 18:00:18.931149 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.931120 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/093a5e7e-2513-47f3-a742-cce2574ac917-maas-api-service-ca" pod "093a5e7e-2513-47f3-a742-cce2574ac917" (UID: "093a5e7e-2513-47f3-a742-cce2574ac917"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:00:18.931539 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.931517 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/093a5e7e-2513-47f3-a742-cce2574ac917-maas-api-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:00:18.931619 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.931606 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889720-t7vtz"] Oct 30 18:00:18.933219 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.933198 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093a5e7e-2513-47f3-a742-cce2574ac917-kube-api-access-6zv75" pod "093a5e7e-2513-47f3-a742-cce2574ac917" (UID: "093a5e7e-2513-47f3-a742-cce2574ac917"). InnerVolumeSpecName "kube-api-access-6zv75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:00:18.965114 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.965088 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp"] Oct 30 18:00:18.969757 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:18.969730 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a23c98-29889720-wrchp"] Oct 30 18:00:19.033490 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:19.033455 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zv75\" (UniqueName: \"kubernetes.io/projected/093a5e7e-2513-47f3-a742-cce2574ac917-kube-api-access-6zv75\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:00:19.698352 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:19.698317 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52a3f74-l9swh" event={"ID":"5e43b107-9066-4368-a04a-8437c7b4201d","Type":"ContainerStarted","Data":"3bbc8ff13be2e68bfe181e89ec68284fa26f57944b4d09aed551229ce277c1f6"} Oct 30 18:00:19.701482 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:19.701451 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-6869fbf48b-jwbm4" event={"ID":"c18e28aa-2d36-480c-9284-4b3b7212539e","Type":"ContainerStarted","Data":"635a5110c3d6a4a9ceca52ba86dd5cac308e573e8def9fb019a861e1b10aa35c"} Oct 30 18:00:19.701812 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:19.701793 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-6869fbf48b-jwbm4" Oct 30 18:00:19.704727 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:19.704707 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88" Oct 30 18:00:19.705303 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:19.705273 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88" event={"ID":"093a5e7e-2513-47f3-a742-cce2574ac917","Type":"ContainerDied","Data":"57bfa982f991e0be11f9035e5f717c62baa18d7b5aa3472d19f8629cfd20ac56"} Oct 30 18:00:19.705390 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:19.705348 4965 scope.go:122] "RemoveContainer" containerID="5e29500b1b48c54f93e157fb42c9ba5004812476290340304dac857f0a64782e" Oct 30 18:00:19.712093 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:19.712066 4965 generic.go:356] "Generic (PLEG): container finished" podID="5e9c7d6c-3ded-495e-809b-5ffccf77b95a" containerID="7af310214cb7a0d685c7d0cb9e15d7b114148efa43bf147e557e0f1613cc0a9f" exitCode=60 Oct 30 18:00:19.712208 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:19.712158 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw" event={"ID":"5e9c7d6c-3ded-495e-809b-5ffccf77b95a","Type":"ContainerDied","Data":"7af310214cb7a0d685c7d0cb9e15d7b114148efa43bf147e557e0f1613cc0a9f"} Oct 30 18:00:19.712662 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:19.712595 4965 scope.go:122] "RemoveContainer" containerID="7af310214cb7a0d685c7d0cb9e15d7b114148efa43bf147e557e0f1613cc0a9f" Oct 30 18:00:19.713922 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:00:19.713011 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw_odh-ai-gateway-infra(5e9c7d6c-3ded-495e-809b-5ffccf77b95a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw" podUID="5e9c7d6c-3ded-495e-809b-5ffccf77b95a" Oct 30 18:00:19.718023 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:19.717786 4965 generic.go:356] "Generic (PLEG): container finished" podID="35173211-ac01-43fa-885c-b148a4f81ae4" containerID="0027bd1942c98b158d32ac1b5d61e296207154c7207ae8a61ffdc8ca0d739070" exitCode=60 Oct 30 18:00:19.718124 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:19.718030 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf" event={"ID":"35173211-ac01-43fa-885c-b148a4f81ae4","Type":"ContainerDied","Data":"0027bd1942c98b158d32ac1b5d61e296207154c7207ae8a61ffdc8ca0d739070"} Oct 30 18:00:19.718594 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:19.718571 4965 scope.go:122] "RemoveContainer" containerID="0027bd1942c98b158d32ac1b5d61e296207154c7207ae8a61ffdc8ca0d739070" Oct 30 18:00:19.719018 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:00:19.718998 4965 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf_odh-ai-gateway-infra(35173211-ac01-43fa-885c-b148a4f81ae4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf" podUID="35173211-ac01-43fa-885c-b148a4f81ae4" Oct 30 18:00:19.719812 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:19.719792 4965 scope.go:122] "RemoveContainer" containerID="c22612339c3c684db5c489e1ec9c33fbd113d367d673dd7e8fe00e3e49142923" Oct 30 18:00:19.736521 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:19.736498 4965 scope.go:122] "RemoveContainer" containerID="a5fbb0b7c22e0cc229b1e28c15b9ab08f2d4fd158d0d153e2ad6ccc30b203c23" Oct 30 18:00:19.796913 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:19.795750 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52a3f74-l9swh" podStartSLOduration=2.795732427 podStartE2EDuration="2.795732427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:19.794320735 +0000 UTC m=+2877.846420689" watchObservedRunningTime="2026-10-30 18:00:19.795732427 +0000 UTC m=+2877.847832359" Oct 30 18:00:19.829747 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:19.829712 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw"] Oct 30 18:00:19.837703 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:19.837645 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-6869fbf48b-jwbm4" podStartSLOduration=2.837628272 podStartE2EDuration="2.837628272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:19.834976066 +0000 UTC m=+2877.887076030" watchObservedRunningTime="2026-10-30 18:00:19.837628272 +0000 UTC m=+2877.889728204" Oct 30 18:00:19.877054 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:19.877025 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88"] Oct 30 18:00:19.881318 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:19.881292 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-01b925-29889720-n6b88"] Oct 30 18:00:19.899532 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:19.899513 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj" Oct 30 18:00:19.996914 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:19.996808 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63f5bed4-8de4-400c-a622-87e37d050d1c-kube-api-access-wtsnc\" (UniqueName: \"kubernetes.io/projected/63f5bed4-8de4-400c-a622-87e37d050d1c-kube-api-access-wtsnc\") pod \"63f5bed4-8de4-400c-a622-87e37d050d1c\" (UID: \"63f5bed4-8de4-400c-a622-87e37d050d1c\") " Oct 30 18:00:19.997567 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:19.997527 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63f5bed4-8de4-400c-a622-87e37d050d1c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63f5bed4-8de4-400c-a622-87e37d050d1c-maas-api-service-ca\") pod \"63f5bed4-8de4-400c-a622-87e37d050d1c\" (UID: \"63f5bed4-8de4-400c-a622-87e37d050d1c\") " Oct 30 18:00:19.998903 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:19.998872 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f5bed4-8de4-400c-a622-87e37d050d1c-maas-api-service-ca" pod "63f5bed4-8de4-400c-a622-87e37d050d1c" (UID: "63f5bed4-8de4-400c-a622-87e37d050d1c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:00:20.009026 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:20.008996 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f5bed4-8de4-400c-a622-87e37d050d1c-kube-api-access-wtsnc" pod "63f5bed4-8de4-400c-a622-87e37d050d1c" (UID: "63f5bed4-8de4-400c-a622-87e37d050d1c"). InnerVolumeSpecName "kube-api-access-wtsnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:00:20.100008 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:20.099972 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63f5bed4-8de4-400c-a622-87e37d050d1c-maas-api-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:00:20.100008 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:20.100008 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtsnc\" (UniqueName: \"kubernetes.io/projected/63f5bed4-8de4-400c-a622-87e37d050d1c-kube-api-access-wtsnc\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:00:20.104613 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:20.104579 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02e21c00-730d-41ec-a53d-ea9f556c2459" path="/var/lib/kubelet/pods/02e21c00-730d-41ec-a53d-ea9f556c2459/volumes" Oct 30 18:00:20.105280 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:20.105257 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="093a5e7e-2513-47f3-a742-cce2574ac917" path="/var/lib/kubelet/pods/093a5e7e-2513-47f3-a742-cce2574ac917/volumes" Oct 30 18:00:20.105898 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:20.105876 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23dc886a-04cf-4d65-b4e5-b9de4c06e228" path="/var/lib/kubelet/pods/23dc886a-04cf-4d65-b4e5-b9de4c06e228/volumes" Oct 30 18:00:20.106981 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:20.106961 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4abe15c2-d8d1-40aa-a0d2-90673380e3ef" path="/var/lib/kubelet/pods/4abe15c2-d8d1-40aa-a0d2-90673380e3ef/volumes" Oct 30 18:00:20.731791 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:20.731757 4965 generic.go:356] "Generic (PLEG): container finished" podID="5e43b107-9066-4368-a04a-8437c7b4201d" containerID="3bbc8ff13be2e68bfe181e89ec68284fa26f57944b4d09aed551229ce277c1f6" exitCode=0 Oct 30 18:00:20.732239 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:20.731835 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52a3f74-l9swh" event={"ID":"5e43b107-9066-4368-a04a-8437c7b4201d","Type":"ContainerDied","Data":"3bbc8ff13be2e68bfe181e89ec68284fa26f57944b4d09aed551229ce277c1f6"} Oct 30 18:00:20.738048 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:20.738019 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj" Oct 30 18:00:20.738430 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:20.738327 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj" event={"ID":"63f5bed4-8de4-400c-a622-87e37d050d1c","Type":"ContainerDied","Data":"8c23cd9ce0c0d21171b2936829b6db8ce13bb1ec07bd62cefe7ca2ff52d25a3e"} Oct 30 18:00:20.738430 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:20.738387 4965 scope.go:122] "RemoveContainer" containerID="9bdbd69e9dc32868f8c12f4f09110e26042d2c9fff7174de168a3d50c07b9e41" Oct 30 18:00:20.814722 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:20.814695 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj"] Oct 30 18:00:20.821601 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:20.820895 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889720-9ttkj"] Oct 30 18:00:20.946967 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:20.946940 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw" Oct 30 18:00:21.020205 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:21.020118 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e9c7d6c-3ded-495e-809b-5ffccf77b95a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e9c7d6c-3ded-495e-809b-5ffccf77b95a-maas-api-service-ca\") pod \"5e9c7d6c-3ded-495e-809b-5ffccf77b95a\" (UID: \"5e9c7d6c-3ded-495e-809b-5ffccf77b95a\") " Oct 30 18:00:21.020371 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:21.020304 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e9c7d6c-3ded-495e-809b-5ffccf77b95a-kube-api-access-22thg\" (UniqueName: \"kubernetes.io/projected/5e9c7d6c-3ded-495e-809b-5ffccf77b95a-kube-api-access-22thg\") pod \"5e9c7d6c-3ded-495e-809b-5ffccf77b95a\" (UID: \"5e9c7d6c-3ded-495e-809b-5ffccf77b95a\") " Oct 30 18:00:21.020555 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:21.020532 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e9c7d6c-3ded-495e-809b-5ffccf77b95a-maas-api-service-ca" pod "5e9c7d6c-3ded-495e-809b-5ffccf77b95a" (UID: "5e9c7d6c-3ded-495e-809b-5ffccf77b95a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:00:21.020987 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:21.020965 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e9c7d6c-3ded-495e-809b-5ffccf77b95a-maas-api-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:00:21.022644 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:21.022615 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e9c7d6c-3ded-495e-809b-5ffccf77b95a-kube-api-access-22thg" pod "5e9c7d6c-3ded-495e-809b-5ffccf77b95a" (UID: "5e9c7d6c-3ded-495e-809b-5ffccf77b95a"). InnerVolumeSpecName "kube-api-access-22thg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:00:21.032827 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:21.032806 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf"] Oct 30 18:00:21.123616 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:21.123575 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22thg\" (UniqueName: \"kubernetes.io/projected/5e9c7d6c-3ded-495e-809b-5ffccf77b95a-kube-api-access-22thg\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:00:21.209456 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:21.209432 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf" Oct 30 18:00:21.325741 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:21.325713 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35173211-ac01-43fa-885c-b148a4f81ae4-kube-api-access-nvjrv\" (UniqueName: \"kubernetes.io/projected/35173211-ac01-43fa-885c-b148a4f81ae4-kube-api-access-nvjrv\") pod \"35173211-ac01-43fa-885c-b148a4f81ae4\" (UID: \"35173211-ac01-43fa-885c-b148a4f81ae4\") " Oct 30 18:00:21.326118 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:21.326097 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35173211-ac01-43fa-885c-b148a4f81ae4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35173211-ac01-43fa-885c-b148a4f81ae4-maas-api-service-ca\") pod \"35173211-ac01-43fa-885c-b148a4f81ae4\" (UID: \"35173211-ac01-43fa-885c-b148a4f81ae4\") " Oct 30 18:00:21.326429 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:21.326406 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35173211-ac01-43fa-885c-b148a4f81ae4-maas-api-service-ca" pod "35173211-ac01-43fa-885c-b148a4f81ae4" (UID: "35173211-ac01-43fa-885c-b148a4f81ae4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:00:21.326668 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:21.326652 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35173211-ac01-43fa-885c-b148a4f81ae4-maas-api-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:00:21.328358 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:21.328337 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35173211-ac01-43fa-885c-b148a4f81ae4-kube-api-access-nvjrv" pod "35173211-ac01-43fa-885c-b148a4f81ae4" (UID: "35173211-ac01-43fa-885c-b148a4f81ae4"). InnerVolumeSpecName "kube-api-access-nvjrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:00:21.427998 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:21.427967 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvjrv\" (UniqueName: \"kubernetes.io/projected/35173211-ac01-43fa-885c-b148a4f81ae4-kube-api-access-nvjrv\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:00:21.753019 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:21.752943 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw" Oct 30 18:00:21.753446 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:21.752938 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw" event={"ID":"5e9c7d6c-3ded-495e-809b-5ffccf77b95a","Type":"ContainerDied","Data":"38b4b69ae371d8f7fbee83f92cb206663ed5d0c3a73bb72fe27d23015323d9a8"} Oct 30 18:00:21.753446 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:21.753071 4965 scope.go:122] "RemoveContainer" containerID="7af310214cb7a0d685c7d0cb9e15d7b114148efa43bf147e557e0f1613cc0a9f" Oct 30 18:00:21.755844 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:21.755819 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf" Oct 30 18:00:21.756025 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:21.755848 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf" event={"ID":"35173211-ac01-43fa-885c-b148a4f81ae4","Type":"ContainerDied","Data":"c9e68dc1a9077152f22b6f8b346a8b5a2cea9f7bfbdedb0ee2e0cc5e56e64a51"} Oct 30 18:00:21.764990 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:21.764915 4965 scope.go:122] "RemoveContainer" containerID="0027bd1942c98b158d32ac1b5d61e296207154c7207ae8a61ffdc8ca0d739070" Oct 30 18:00:21.808243 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:21.808194 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf"] Oct 30 18:00:21.813033 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:21.813006 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889720-xhlpf"] Oct 30 18:00:21.847627 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:21.847575 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw"] Oct 30 18:00:21.858773 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:21.858742 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-738845-29889720-cg2tw"] Oct 30 18:00:21.939747 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:21.939723 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52a3f74-l9swh" Oct 30 18:00:22.037809 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:22.037722 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e43b107-9066-4368-a04a-8437c7b4201d-kube-api-access-vxkj9\" (UniqueName: \"kubernetes.io/projected/5e43b107-9066-4368-a04a-8437c7b4201d-kube-api-access-vxkj9\") pod \"5e43b107-9066-4368-a04a-8437c7b4201d\" (UID: \"5e43b107-9066-4368-a04a-8437c7b4201d\") " Oct 30 18:00:22.038008 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:22.037806 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e43b107-9066-4368-a04a-8437c7b4201d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e43b107-9066-4368-a04a-8437c7b4201d-maas-api-xks-ca\") pod \"5e43b107-9066-4368-a04a-8437c7b4201d\" (UID: \"5e43b107-9066-4368-a04a-8437c7b4201d\") " Oct 30 18:00:22.038075 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:22.038053 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e43b107-9066-4368-a04a-8437c7b4201d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e43b107-9066-4368-a04a-8437c7b4201d-maas-api-openshift-ca\") pod \"5e43b107-9066-4368-a04a-8437c7b4201d\" (UID: \"5e43b107-9066-4368-a04a-8437c7b4201d\") " Oct 30 18:00:22.038520 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:22.038490 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e43b107-9066-4368-a04a-8437c7b4201d-maas-api-openshift-ca" pod "5e43b107-9066-4368-a04a-8437c7b4201d" (UID: "5e43b107-9066-4368-a04a-8437c7b4201d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:00:22.038728 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:22.038707 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e43b107-9066-4368-a04a-8437c7b4201d-maas-api-openshift-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:00:22.040141 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:22.040118 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e43b107-9066-4368-a04a-8437c7b4201d-kube-api-access-vxkj9" pod "5e43b107-9066-4368-a04a-8437c7b4201d" (UID: "5e43b107-9066-4368-a04a-8437c7b4201d"). InnerVolumeSpecName "kube-api-access-vxkj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:00:22.040729 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:22.040714 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e43b107-9066-4368-a04a-8437c7b4201d-maas-api-xks-ca" pod "5e43b107-9066-4368-a04a-8437c7b4201d" (UID: "5e43b107-9066-4368-a04a-8437c7b4201d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:00:22.101135 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:22.101102 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35173211-ac01-43fa-885c-b148a4f81ae4" path="/var/lib/kubelet/pods/35173211-ac01-43fa-885c-b148a4f81ae4/volumes" Oct 30 18:00:22.101731 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:22.101704 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e9c7d6c-3ded-495e-809b-5ffccf77b95a" path="/var/lib/kubelet/pods/5e9c7d6c-3ded-495e-809b-5ffccf77b95a/volumes" Oct 30 18:00:22.102188 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:22.102173 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63f5bed4-8de4-400c-a622-87e37d050d1c" path="/var/lib/kubelet/pods/63f5bed4-8de4-400c-a622-87e37d050d1c/volumes" Oct 30 18:00:22.141062 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:22.140889 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxkj9\" (UniqueName: \"kubernetes.io/projected/5e43b107-9066-4368-a04a-8437c7b4201d-kube-api-access-vxkj9\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:00:22.142481 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:22.141260 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e43b107-9066-4368-a04a-8437c7b4201d-maas-api-xks-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:00:22.767185 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:22.767138 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52a3f74-l9swh" event={"ID":"5e43b107-9066-4368-a04a-8437c7b4201d","Type":"ContainerDied","Data":"94f85682dc54250b94ad91fe931f00704ada991b384d53be4e6bc98045c4ec62"} Oct 30 18:00:22.767185 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:22.767175 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52a3f74-l9swh" Oct 30 18:00:22.767609 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:22.767245 4965 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94f85682dc54250b94ad91fe931f00704ada991b384d53be4e6bc98045c4ec62" Oct 30 18:00:30.728327 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.728290 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-844694479f-vx68s"] Oct 30 18:00:30.729103 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729086 4965 cpu_manager.go:414] "containerMap: removing container" podUID="02e21c00-730d-41ec-a53d-ea9f556c2459" containerName="cleanup" Oct 30 18:00:30.729161 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729104 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02e21c00-730d-41ec-a53d-ea9f556c2459" containerName="cleanup" Oct 30 18:00:30.729161 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729113 4965 cpu_manager.go:414] "containerMap: removing container" podUID="4abe15c2-d8d1-40aa-a0d2-90673380e3ef" containerName="cleanup" Oct 30 18:00:30.729161 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729118 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4abe15c2-d8d1-40aa-a0d2-90673380e3ef" containerName="cleanup" Oct 30 18:00:30.729161 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729127 4965 cpu_manager.go:414] "containerMap: removing container" podUID="23dc886a-04cf-4d65-b4e5-b9de4c06e228" containerName="cleanup" Oct 30 18:00:30.729161 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729132 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23dc886a-04cf-4d65-b4e5-b9de4c06e228" containerName="cleanup" Oct 30 18:00:30.729161 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729142 4965 cpu_manager.go:414] "containerMap: removing container" podUID="35173211-ac01-43fa-885c-b148a4f81ae4" containerName="cleanup" Oct 30 18:00:30.729161 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729146 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35173211-ac01-43fa-885c-b148a4f81ae4" containerName="cleanup" Oct 30 18:00:30.729161 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729154 4965 cpu_manager.go:414] "containerMap: removing container" podUID="4abe15c2-d8d1-40aa-a0d2-90673380e3ef" containerName="cleanup" Oct 30 18:00:30.729161 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729159 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4abe15c2-d8d1-40aa-a0d2-90673380e3ef" containerName="cleanup" Oct 30 18:00:30.729447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729165 4965 cpu_manager.go:414] "containerMap: removing container" podUID="5e9c7d6c-3ded-495e-809b-5ffccf77b95a" containerName="cleanup" Oct 30 18:00:30.729447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729170 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e9c7d6c-3ded-495e-809b-5ffccf77b95a" containerName="cleanup" Oct 30 18:00:30.729447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729179 4965 cpu_manager.go:414] "containerMap: removing container" podUID="5e43b107-9066-4368-a04a-8437c7b4201d" containerName="revoke-keys" Oct 30 18:00:30.729447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729185 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e43b107-9066-4368-a04a-8437c7b4201d" containerName="revoke-keys" Oct 30 18:00:30.729447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729197 4965 cpu_manager.go:414] "containerMap: removing container" podUID="093a5e7e-2513-47f3-a742-cce2574ac917" containerName="cleanup" Oct 30 18:00:30.729447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729202 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="093a5e7e-2513-47f3-a742-cce2574ac917" containerName="cleanup" Oct 30 18:00:30.729447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729212 4965 cpu_manager.go:414] "containerMap: removing container" podUID="23dc886a-04cf-4d65-b4e5-b9de4c06e228" containerName="cleanup" Oct 30 18:00:30.729447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729216 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23dc886a-04cf-4d65-b4e5-b9de4c06e228" containerName="cleanup" Oct 30 18:00:30.729447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729234 4965 cpu_manager.go:414] "containerMap: removing container" podUID="63f5bed4-8de4-400c-a622-87e37d050d1c" containerName="cleanup" Oct 30 18:00:30.729447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729238 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63f5bed4-8de4-400c-a622-87e37d050d1c" containerName="cleanup" Oct 30 18:00:30.729447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729248 4965 cpu_manager.go:414] "containerMap: removing container" podUID="63f5bed4-8de4-400c-a622-87e37d050d1c" containerName="cleanup" Oct 30 18:00:30.729447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729253 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63f5bed4-8de4-400c-a622-87e37d050d1c" containerName="cleanup" Oct 30 18:00:30.729447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729260 4965 cpu_manager.go:414] "containerMap: removing container" podUID="63f5bed4-8de4-400c-a622-87e37d050d1c" containerName="cleanup" Oct 30 18:00:30.729447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729264 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63f5bed4-8de4-400c-a622-87e37d050d1c" containerName="cleanup" Oct 30 18:00:30.729447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729275 4965 cpu_manager.go:414] "containerMap: removing container" podUID="5e9c7d6c-3ded-495e-809b-5ffccf77b95a" containerName="cleanup" Oct 30 18:00:30.729447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729279 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e9c7d6c-3ded-495e-809b-5ffccf77b95a" containerName="cleanup" Oct 30 18:00:30.729447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729292 4965 cpu_manager.go:414] "containerMap: removing container" podUID="35173211-ac01-43fa-885c-b148a4f81ae4" containerName="cleanup" Oct 30 18:00:30.729447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729296 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35173211-ac01-43fa-885c-b148a4f81ae4" containerName="cleanup" Oct 30 18:00:30.729447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729312 4965 cpu_manager.go:414] "containerMap: removing container" podUID="02e21c00-730d-41ec-a53d-ea9f556c2459" containerName="cleanup" Oct 30 18:00:30.729447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729317 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02e21c00-730d-41ec-a53d-ea9f556c2459" containerName="cleanup" Oct 30 18:00:30.729447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729323 4965 cpu_manager.go:414] "containerMap: removing container" podUID="093a5e7e-2513-47f3-a742-cce2574ac917" containerName="cleanup" Oct 30 18:00:30.729447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729327 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="093a5e7e-2513-47f3-a742-cce2574ac917" containerName="cleanup" Oct 30 18:00:30.729447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729430 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e43b107-9066-4368-a04a-8437c7b4201d" containerName="revoke-keys" Oct 30 18:00:30.729447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729438 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="093a5e7e-2513-47f3-a742-cce2574ac917" containerName="cleanup" Oct 30 18:00:30.729447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729445 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="35173211-ac01-43fa-885c-b148a4f81ae4" containerName="cleanup" Oct 30 18:00:30.729447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729451 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="23dc886a-04cf-4d65-b4e5-b9de4c06e228" containerName="cleanup" Oct 30 18:00:30.729447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729456 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="093a5e7e-2513-47f3-a742-cce2574ac917" containerName="cleanup" Oct 30 18:00:30.730338 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729464 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="63f5bed4-8de4-400c-a622-87e37d050d1c" containerName="cleanup" Oct 30 18:00:30.730338 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729471 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="63f5bed4-8de4-400c-a622-87e37d050d1c" containerName="cleanup" Oct 30 18:00:30.730338 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729476 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="4abe15c2-d8d1-40aa-a0d2-90673380e3ef" containerName="cleanup" Oct 30 18:00:30.730338 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729480 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="63f5bed4-8de4-400c-a622-87e37d050d1c" containerName="cleanup" Oct 30 18:00:30.730338 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729486 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e9c7d6c-3ded-495e-809b-5ffccf77b95a" containerName="cleanup" Oct 30 18:00:30.730338 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729493 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="35173211-ac01-43fa-885c-b148a4f81ae4" containerName="cleanup" Oct 30 18:00:30.730338 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729498 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="02e21c00-730d-41ec-a53d-ea9f556c2459" containerName="cleanup" Oct 30 18:00:30.730338 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729505 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="02e21c00-730d-41ec-a53d-ea9f556c2459" containerName="cleanup" Oct 30 18:00:30.730338 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729512 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="02e21c00-730d-41ec-a53d-ea9f556c2459" containerName="cleanup" Oct 30 18:00:30.730338 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729516 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="4abe15c2-d8d1-40aa-a0d2-90673380e3ef" containerName="cleanup" Oct 30 18:00:30.730338 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729522 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="23dc886a-04cf-4d65-b4e5-b9de4c06e228" containerName="cleanup" Oct 30 18:00:30.730338 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729527 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="4abe15c2-d8d1-40aa-a0d2-90673380e3ef" containerName="cleanup" Oct 30 18:00:30.730338 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.729533 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e9c7d6c-3ded-495e-809b-5ffccf77b95a" containerName="cleanup" Oct 30 18:00:30.735818 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.735798 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-844694479f-vx68s" Oct 30 18:00:30.753468 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.753445 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-6869fbf48b-jwbm4" Oct 30 18:00:30.754045 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.754023 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-844694479f-vx68s"] Oct 30 18:00:30.763910 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.760697 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b5cbb4\"" Oct 30 18:00:30.763910 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.760845 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b5cbb4-dockercfg-mzz4l\"" Oct 30 18:00:30.864885 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.863887 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e012d8-59ad-4aad-818d-b48aa6b35428-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b5cbb4-844694479f-vx68s\" (UID: \"e2e012d8-59ad-4aad-818d-b48aa6b35428\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-844694479f-vx68s" Oct 30 18:00:30.864885 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.864286 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5wb6\" (UniqueName: \"kubernetes.io/projected/e2e012d8-59ad-4aad-818d-b48aa6b35428-kube-api-access-w5wb6\") pod \"payload-pre-processing-e2e-ipp-b-b5cbb4-844694479f-vx68s\" (UID: \"e2e012d8-59ad-4aad-818d-b48aa6b35428\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-844694479f-vx68s" Oct 30 18:00:31.002004 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.990435 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5wb6\" (UniqueName: \"kubernetes.io/projected/e2e012d8-59ad-4aad-818d-b48aa6b35428-kube-api-access-w5wb6\") pod \"payload-pre-processing-e2e-ipp-b-b5cbb4-844694479f-vx68s\" (UID: \"e2e012d8-59ad-4aad-818d-b48aa6b35428\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-844694479f-vx68s" Oct 30 18:00:31.002004 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.998007 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e012d8-59ad-4aad-818d-b48aa6b35428-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b5cbb4-844694479f-vx68s\" (UID: \"e2e012d8-59ad-4aad-818d-b48aa6b35428\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-844694479f-vx68s" Oct 30 18:00:31.002004 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:30.999295 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e012d8-59ad-4aad-818d-b48aa6b35428-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b5cbb4-844694479f-vx68s\" (UID: \"e2e012d8-59ad-4aad-818d-b48aa6b35428\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-844694479f-vx68s" Oct 30 18:00:31.050728 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:31.041339 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5wb6\" (UniqueName: \"kubernetes.io/projected/e2e012d8-59ad-4aad-818d-b48aa6b35428-kube-api-access-w5wb6\") pod \"payload-pre-processing-e2e-ipp-b-b5cbb4-844694479f-vx68s\" (UID: \"e2e012d8-59ad-4aad-818d-b48aa6b35428\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-844694479f-vx68s" Oct 30 18:00:31.053978 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:31.053692 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-844694479f-vx68s" Oct 30 18:00:31.382903 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:31.382839 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-844694479f-vx68s"] Oct 30 18:00:31.383897 ip-10-0-139-222 kubenswrapper[4965]: W1030 18:00:31.383838 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e012d8_59ad_4aad_818d_b48aa6b35428.slice/crio-bf40098e343598371468c30fd3789df5f11ab3c71fdfb5bc42f9c2cfb8144a8c WatchSource:0}: Error finding container bf40098e343598371468c30fd3789df5f11ab3c71fdfb5bc42f9c2cfb8144a8c: Status 404 returned error can't find the container with id bf40098e343598371468c30fd3789df5f11ab3c71fdfb5bc42f9c2cfb8144a8c Oct 30 18:00:31.968479 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:31.968422 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-844694479f-vx68s" event={"ID":"e2e012d8-59ad-4aad-818d-b48aa6b35428","Type":"ContainerStarted","Data":"ee11c49174078f979f6e56b5b02adbcf5f47ca4f7919b9585ed7645f417c7aae"} Oct 30 18:00:31.968997 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:31.968509 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-844694479f-vx68s" Oct 30 18:00:31.968997 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:31.968526 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-844694479f-vx68s" event={"ID":"e2e012d8-59ad-4aad-818d-b48aa6b35428","Type":"ContainerStarted","Data":"bf40098e343598371468c30fd3789df5f11ab3c71fdfb5bc42f9c2cfb8144a8c"} Oct 30 18:00:32.009261 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:32.009198 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-844694479f-vx68s" podStartSLOduration=2.009179675 podStartE2EDuration="2.009179675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:32.008126553 +0000 UTC m=+2890.060226485" watchObservedRunningTime="2026-10-30 18:00:32.009179675 +0000 UTC m=+2890.061279610" Oct 30 18:00:34.589327 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:34.589287 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fab25f-66d654dd9d-nv9mt"] Oct 30 18:00:34.590313 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:34.590289 4965 cpu_manager.go:414] "containerMap: removing container" podUID="093a5e7e-2513-47f3-a742-cce2574ac917" containerName="cleanup" Oct 30 18:00:34.590313 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:34.590312 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="093a5e7e-2513-47f3-a742-cce2574ac917" containerName="cleanup" Oct 30 18:00:34.590491 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:34.590336 4965 cpu_manager.go:414] "containerMap: removing container" podUID="23dc886a-04cf-4d65-b4e5-b9de4c06e228" containerName="cleanup" Oct 30 18:00:34.590491 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:34.590343 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23dc886a-04cf-4d65-b4e5-b9de4c06e228" containerName="cleanup" Oct 30 18:00:34.590491 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:34.590383 4965 cpu_manager.go:414] "containerMap: removing container" podUID="5e9c7d6c-3ded-495e-809b-5ffccf77b95a" containerName="cleanup" Oct 30 18:00:34.590491 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:34.590391 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e9c7d6c-3ded-495e-809b-5ffccf77b95a" containerName="cleanup" Oct 30 18:00:34.590491 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:34.590444 4965 cpu_manager.go:414] "containerMap: removing container" podUID="02e21c00-730d-41ec-a53d-ea9f556c2459" containerName="cleanup" Oct 30 18:00:34.590491 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:34.590452 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02e21c00-730d-41ec-a53d-ea9f556c2459" containerName="cleanup" Oct 30 18:00:34.590491 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:34.590469 4965 cpu_manager.go:414] "containerMap: removing container" podUID="35173211-ac01-43fa-885c-b148a4f81ae4" containerName="cleanup" Oct 30 18:00:34.590491 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:34.590476 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35173211-ac01-43fa-885c-b148a4f81ae4" containerName="cleanup" Oct 30 18:00:34.590491 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:34.590488 4965 cpu_manager.go:414] "containerMap: removing container" podUID="4abe15c2-d8d1-40aa-a0d2-90673380e3ef" containerName="cleanup" Oct 30 18:00:34.590985 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:34.590494 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4abe15c2-d8d1-40aa-a0d2-90673380e3ef" containerName="cleanup" Oct 30 18:00:34.590985 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:34.590675 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e9c7d6c-3ded-495e-809b-5ffccf77b95a" containerName="cleanup" Oct 30 18:00:34.590985 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:34.590694 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="35173211-ac01-43fa-885c-b148a4f81ae4" containerName="cleanup" Oct 30 18:00:34.590985 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:34.590717 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="23dc886a-04cf-4d65-b4e5-b9de4c06e228" containerName="cleanup" Oct 30 18:00:34.590985 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:34.590726 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="093a5e7e-2513-47f3-a742-cce2574ac917" containerName="cleanup" Oct 30 18:00:34.594949 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:34.594927 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-66d654dd9d-nv9mt" Oct 30 18:00:34.613909 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:34.613882 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fab25f-66d654dd9d-nv9mt"] Oct 30 18:00:34.619641 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:34.619618 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-fab25f-dockercfg-h8xcz\"" Oct 30 18:00:34.619764 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:34.619714 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-fab25f\"" Oct 30 18:00:34.712911 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:34.712847 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhv5f\" (UniqueName: \"kubernetes.io/projected/78b12271-78e7-4b9f-98be-cc3d6b736708-kube-api-access-zhv5f\") pod \"payload-processing-e2e-ipp-a-fab25f-66d654dd9d-nv9mt\" (UID: \"78b12271-78e7-4b9f-98be-cc3d6b736708\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-66d654dd9d-nv9mt" Oct 30 18:00:34.713075 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:34.712990 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78b12271-78e7-4b9f-98be-cc3d6b736708-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fab25f-66d654dd9d-nv9mt\" (UID: \"78b12271-78e7-4b9f-98be-cc3d6b736708\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-66d654dd9d-nv9mt" Oct 30 18:00:34.814157 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:34.814124 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhv5f\" (UniqueName: \"kubernetes.io/projected/78b12271-78e7-4b9f-98be-cc3d6b736708-kube-api-access-zhv5f\") pod \"payload-processing-e2e-ipp-a-fab25f-66d654dd9d-nv9mt\" (UID: \"78b12271-78e7-4b9f-98be-cc3d6b736708\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-66d654dd9d-nv9mt" Oct 30 18:00:34.814346 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:34.814194 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78b12271-78e7-4b9f-98be-cc3d6b736708-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fab25f-66d654dd9d-nv9mt\" (UID: \"78b12271-78e7-4b9f-98be-cc3d6b736708\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-66d654dd9d-nv9mt" Oct 30 18:00:34.814953 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:34.814934 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78b12271-78e7-4b9f-98be-cc3d6b736708-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fab25f-66d654dd9d-nv9mt\" (UID: \"78b12271-78e7-4b9f-98be-cc3d6b736708\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-66d654dd9d-nv9mt" Oct 30 18:00:34.837405 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:34.837379 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhv5f\" (UniqueName: \"kubernetes.io/projected/78b12271-78e7-4b9f-98be-cc3d6b736708-kube-api-access-zhv5f\") pod \"payload-processing-e2e-ipp-a-fab25f-66d654dd9d-nv9mt\" (UID: \"78b12271-78e7-4b9f-98be-cc3d6b736708\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-66d654dd9d-nv9mt" Oct 30 18:00:34.949945 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:34.940724 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-66d654dd9d-nv9mt" Oct 30 18:00:35.268635 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:35.268604 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fab25f-66d654dd9d-nv9mt"] Oct 30 18:00:35.268956 ip-10-0-139-222 kubenswrapper[4965]: W1030 18:00:35.268932 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78b12271_78e7_4b9f_98be_cc3d6b736708.slice/crio-15709efa76cc2f24554d1a666ec5094c89ce89dd6a8bb25cec14e121c2f2e812 WatchSource:0}: Error finding container 15709efa76cc2f24554d1a666ec5094c89ce89dd6a8bb25cec14e121c2f2e812: Status 404 returned error can't find the container with id 15709efa76cc2f24554d1a666ec5094c89ce89dd6a8bb25cec14e121c2f2e812 Oct 30 18:00:35.998563 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:00:35.998529 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:00:36.008316 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:36.008280 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-66d654dd9d-nv9mt" event={"ID":"78b12271-78e7-4b9f-98be-cc3d6b736708","Type":"ContainerStarted","Data":"94de12c06ff2453ff5538008824801c81958cd66240ea926605e8c4459784f56"} Oct 30 18:00:36.008485 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:36.008376 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-66d654dd9d-nv9mt" Oct 30 18:00:36.008485 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:36.008395 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-66d654dd9d-nv9mt" event={"ID":"78b12271-78e7-4b9f-98be-cc3d6b736708","Type":"ContainerStarted","Data":"15709efa76cc2f24554d1a666ec5094c89ce89dd6a8bb25cec14e121c2f2e812"} Oct 30 18:00:36.052043 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:36.051996 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-66d654dd9d-nv9mt" podStartSLOduration=2.05198383 podStartE2EDuration="2.05198383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:36.04904463 +0000 UTC m=+2894.101144562" watchObservedRunningTime="2026-10-30 18:00:36.05198383 +0000 UTC m=+2894.104083760" Oct 30 18:00:42.979122 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:42.979093 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-844694479f-vx68s" Oct 30 18:00:47.019492 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:00:47.019465 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-66d654dd9d-nv9mt" Oct 30 18:01:05.998557 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:01:05.998473 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:01:19.449796 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:19.449759 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:01:35.998291 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:01:35.998242 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:01:57.463588 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:57.463537 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-7b54dd4f5-qtdnz"] Oct 30 18:01:57.471096 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:57.471071 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-7b54dd4f5-qtdnz"] Oct 30 18:01:57.471611 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:57.471589 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-7b54dd4f5-qtdnz" Oct 30 18:01:57.498091 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:57.498060 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e927d7-dockercfg-wrd6c\"" Oct 30 18:01:57.499659 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:57.498564 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e927d7\"" Oct 30 18:01:57.524856 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:57.524817 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a23856b6-65d8-489d-9bae-299672aa44d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e927d7-7b54dd4f5-qtdnz\" (UID: \"a23856b6-65d8-489d-9bae-299672aa44d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-7b54dd4f5-qtdnz" Oct 30 18:01:57.525070 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:57.524979 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtlds\" (UniqueName: \"kubernetes.io/projected/a23856b6-65d8-489d-9bae-299672aa44d2-kube-api-access-vtlds\") pod \"payload-pre-processing-e2e-ipp-b-e927d7-7b54dd4f5-qtdnz\" (UID: \"a23856b6-65d8-489d-9bae-299672aa44d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-7b54dd4f5-qtdnz" Oct 30 18:01:57.545064 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:57.545026 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e927d7-5f4dcd6d56-kq9wj"] Oct 30 18:01:57.552318 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:57.552155 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-5f4dcd6d56-kq9wj" Oct 30 18:01:57.566029 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:57.565993 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e927d7-5f4dcd6d56-kq9wj"] Oct 30 18:01:57.626130 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:57.626089 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a23856b6-65d8-489d-9bae-299672aa44d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e927d7-7b54dd4f5-qtdnz\" (UID: \"a23856b6-65d8-489d-9bae-299672aa44d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-7b54dd4f5-qtdnz" Oct 30 18:01:57.626312 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:57.626225 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtlds\" (UniqueName: \"kubernetes.io/projected/a23856b6-65d8-489d-9bae-299672aa44d2-kube-api-access-vtlds\") pod \"payload-pre-processing-e2e-ipp-b-e927d7-7b54dd4f5-qtdnz\" (UID: \"a23856b6-65d8-489d-9bae-299672aa44d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-7b54dd4f5-qtdnz" Oct 30 18:01:57.626383 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:57.626310 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rqr7\" (UniqueName: \"kubernetes.io/projected/e84b153b-396c-4135-bd60-a92cac969337-kube-api-access-6rqr7\") pod \"payload-processing-e2e-ipp-b-e927d7-5f4dcd6d56-kq9wj\" (UID: \"e84b153b-396c-4135-bd60-a92cac969337\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-5f4dcd6d56-kq9wj" Oct 30 18:01:57.626487 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:57.626467 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e84b153b-396c-4135-bd60-a92cac969337-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e927d7-5f4dcd6d56-kq9wj\" (UID: \"e84b153b-396c-4135-bd60-a92cac969337\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-5f4dcd6d56-kq9wj" Oct 30 18:01:57.627067 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:57.627042 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a23856b6-65d8-489d-9bae-299672aa44d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e927d7-7b54dd4f5-qtdnz\" (UID: \"a23856b6-65d8-489d-9bae-299672aa44d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-7b54dd4f5-qtdnz" Oct 30 18:01:57.649896 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:57.649856 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtlds\" (UniqueName: \"kubernetes.io/projected/a23856b6-65d8-489d-9bae-299672aa44d2-kube-api-access-vtlds\") pod \"payload-pre-processing-e2e-ipp-b-e927d7-7b54dd4f5-qtdnz\" (UID: \"a23856b6-65d8-489d-9bae-299672aa44d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-7b54dd4f5-qtdnz" Oct 30 18:01:57.728304 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:57.728210 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rqr7\" (UniqueName: \"kubernetes.io/projected/e84b153b-396c-4135-bd60-a92cac969337-kube-api-access-6rqr7\") pod \"payload-processing-e2e-ipp-b-e927d7-5f4dcd6d56-kq9wj\" (UID: \"e84b153b-396c-4135-bd60-a92cac969337\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-5f4dcd6d56-kq9wj" Oct 30 18:01:57.728466 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:57.728379 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e84b153b-396c-4135-bd60-a92cac969337-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e927d7-5f4dcd6d56-kq9wj\" (UID: \"e84b153b-396c-4135-bd60-a92cac969337\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-5f4dcd6d56-kq9wj" Oct 30 18:01:57.729103 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:57.729083 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e84b153b-396c-4135-bd60-a92cac969337-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e927d7-5f4dcd6d56-kq9wj\" (UID: \"e84b153b-396c-4135-bd60-a92cac969337\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-5f4dcd6d56-kq9wj" Oct 30 18:01:57.751634 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:57.751606 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rqr7\" (UniqueName: \"kubernetes.io/projected/e84b153b-396c-4135-bd60-a92cac969337-kube-api-access-6rqr7\") pod \"payload-processing-e2e-ipp-b-e927d7-5f4dcd6d56-kq9wj\" (UID: \"e84b153b-396c-4135-bd60-a92cac969337\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-5f4dcd6d56-kq9wj" Oct 30 18:01:57.810655 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:57.810621 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-7b54dd4f5-qtdnz" Oct 30 18:01:57.872660 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:57.872630 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-5f4dcd6d56-kq9wj" Oct 30 18:01:58.143833 ip-10-0-139-222 kubenswrapper[4965]: W1030 18:01:58.134026 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda23856b6_65d8_489d_9bae_299672aa44d2.slice/crio-350153c5f6e928f440c41347b0e65c25435eeb2909f9c3b908bd9761f9ea9d53 WatchSource:0}: Error finding container 350153c5f6e928f440c41347b0e65c25435eeb2909f9c3b908bd9761f9ea9d53: Status 404 returned error can't find the container with id 350153c5f6e928f440c41347b0e65c25435eeb2909f9c3b908bd9761f9ea9d53 Oct 30 18:01:58.167806 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:58.167771 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-7b54dd4f5-qtdnz"] Oct 30 18:01:58.198943 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:58.198922 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e927d7-5f4dcd6d56-kq9wj"] Oct 30 18:01:58.202339 ip-10-0-139-222 kubenswrapper[4965]: W1030 18:01:58.202317 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode84b153b_396c_4135_bd60_a92cac969337.slice/crio-b80591768fc39715c2e4c4af3c1a616083a3f83f40d9f9cbda79123f01f580b1 WatchSource:0}: Error finding container b80591768fc39715c2e4c4af3c1a616083a3f83f40d9f9cbda79123f01f580b1: Status 404 returned error can't find the container with id b80591768fc39715c2e4c4af3c1a616083a3f83f40d9f9cbda79123f01f580b1 Oct 30 18:01:58.923542 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:58.923503 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-7b54dd4f5-qtdnz" event={"ID":"a23856b6-65d8-489d-9bae-299672aa44d2","Type":"ContainerStarted","Data":"aed30f9ab6f0961ea9f58bec2c5b2ce775a05620cb36758ded6a344d292f4d19"} Oct 30 18:01:58.924097 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:58.923572 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-7b54dd4f5-qtdnz" event={"ID":"a23856b6-65d8-489d-9bae-299672aa44d2","Type":"ContainerStarted","Data":"350153c5f6e928f440c41347b0e65c25435eeb2909f9c3b908bd9761f9ea9d53"} Oct 30 18:01:58.924097 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:58.923599 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-7b54dd4f5-qtdnz" Oct 30 18:01:58.925225 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:58.925199 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-5f4dcd6d56-kq9wj" event={"ID":"e84b153b-396c-4135-bd60-a92cac969337","Type":"ContainerStarted","Data":"0ec8f0553e377e9102eab0ebd23badf478704ba0eac0fcf9e390c09980dce1dc"} Oct 30 18:01:58.925378 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:58.925239 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-5f4dcd6d56-kq9wj" event={"ID":"e84b153b-396c-4135-bd60-a92cac969337","Type":"ContainerStarted","Data":"b80591768fc39715c2e4c4af3c1a616083a3f83f40d9f9cbda79123f01f580b1"} Oct 30 18:01:58.925378 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:58.925292 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-5f4dcd6d56-kq9wj" Oct 30 18:01:58.967691 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:58.967644 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-7b54dd4f5-qtdnz" podStartSLOduration=1.967629933 podStartE2EDuration="1.967629933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:58.963244934 +0000 UTC m=+2977.015344867" watchObservedRunningTime="2026-10-30 18:01:58.967629933 +0000 UTC m=+2977.019729864" Oct 30 18:01:59.010080 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:01:59.010029 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-5f4dcd6d56-kq9wj" podStartSLOduration=2.010016084 podStartE2EDuration="2.010016084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:59.005955286 +0000 UTC m=+2977.058055244" watchObservedRunningTime="2026-10-30 18:01:59.010016084 +0000 UTC m=+2977.062116038" Oct 30 18:02:03.416394 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:03.416360 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-584a12-6869fbf48b-jwbm4"] Oct 30 18:02:05.998277 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:02:05.998236 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:02:06.944429 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:06.944397 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2"] Oct 30 18:02:06.949367 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:06.949345 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2" Oct 30 18:02:06.988895 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:06.976303 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-f10dda-serving-cert\"" Oct 30 18:02:06.988895 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:06.983833 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2"] Oct 30 18:02:07.106219 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:07.106183 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9c048dc-92d6-46c4-9ada-47b9b5d57e35-metrics-certs\") pod \"maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2\" (UID: \"d9c048dc-92d6-46c4-9ada-47b9b5d57e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2" Oct 30 18:02:07.106636 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:07.106379 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbkl8\" (UniqueName: \"kubernetes.io/projected/d9c048dc-92d6-46c4-9ada-47b9b5d57e35-kube-api-access-xbkl8\") pod \"maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2\" (UID: \"d9c048dc-92d6-46c4-9ada-47b9b5d57e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2" Oct 30 18:02:07.106636 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:07.106410 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9c048dc-92d6-46c4-9ada-47b9b5d57e35-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2\" (UID: \"d9c048dc-92d6-46c4-9ada-47b9b5d57e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2" Oct 30 18:02:07.106636 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:07.106450 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c048dc-92d6-46c4-9ada-47b9b5d57e35-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2\" (UID: \"d9c048dc-92d6-46c4-9ada-47b9b5d57e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2" Oct 30 18:02:07.207821 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:07.207733 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9c048dc-92d6-46c4-9ada-47b9b5d57e35-metrics-certs\") pod \"maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2\" (UID: \"d9c048dc-92d6-46c4-9ada-47b9b5d57e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2" Oct 30 18:02:07.208022 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:07.207895 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbkl8\" (UniqueName: \"kubernetes.io/projected/d9c048dc-92d6-46c4-9ada-47b9b5d57e35-kube-api-access-xbkl8\") pod \"maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2\" (UID: \"d9c048dc-92d6-46c4-9ada-47b9b5d57e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2" Oct 30 18:02:07.208022 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:07.207916 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9c048dc-92d6-46c4-9ada-47b9b5d57e35-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2\" (UID: \"d9c048dc-92d6-46c4-9ada-47b9b5d57e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2" Oct 30 18:02:07.208022 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:07.207948 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c048dc-92d6-46c4-9ada-47b9b5d57e35-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2\" (UID: \"d9c048dc-92d6-46c4-9ada-47b9b5d57e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2" Oct 30 18:02:07.208655 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:07.208625 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c048dc-92d6-46c4-9ada-47b9b5d57e35-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2\" (UID: \"d9c048dc-92d6-46c4-9ada-47b9b5d57e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2" Oct 30 18:02:07.210382 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:07.210356 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9c048dc-92d6-46c4-9ada-47b9b5d57e35-metrics-certs\") pod \"maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2\" (UID: \"d9c048dc-92d6-46c4-9ada-47b9b5d57e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2" Oct 30 18:02:07.210482 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:07.210410 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9c048dc-92d6-46c4-9ada-47b9b5d57e35-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2\" (UID: \"d9c048dc-92d6-46c4-9ada-47b9b5d57e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2" Oct 30 18:02:07.258907 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:07.250097 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbkl8\" (UniqueName: \"kubernetes.io/projected/d9c048dc-92d6-46c4-9ada-47b9b5d57e35-kube-api-access-xbkl8\") pod \"maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2\" (UID: \"d9c048dc-92d6-46c4-9ada-47b9b5d57e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2" Oct 30 18:02:07.269318 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:07.269024 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2" Oct 30 18:02:07.614988 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:07.614957 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2"] Oct 30 18:02:07.615410 ip-10-0-139-222 kubenswrapper[4965]: W1030 18:02:07.615390 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9c048dc_92d6_46c4_9ada_47b9b5d57e35.slice/crio-fc0d8a375c1f7f98d6314a8ec8cefe8755ea9bb92d004f294b166f4c648af109 WatchSource:0}: Error finding container fc0d8a375c1f7f98d6314a8ec8cefe8755ea9bb92d004f294b166f4c648af109: Status 404 returned error can't find the container with id fc0d8a375c1f7f98d6314a8ec8cefe8755ea9bb92d004f294b166f4c648af109 Oct 30 18:02:08.036097 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:08.036061 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2" event={"ID":"d9c048dc-92d6-46c4-9ada-47b9b5d57e35","Type":"ContainerStarted","Data":"07ed131f183e55c3aae4aa5ffadbe84a858ba90b52507f2686b1b0e52d47859a"} Oct 30 18:02:08.036296 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:08.036138 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2" event={"ID":"d9c048dc-92d6-46c4-9ada-47b9b5d57e35","Type":"ContainerStarted","Data":"fc0d8a375c1f7f98d6314a8ec8cefe8755ea9bb92d004f294b166f4c648af109"} Oct 30 18:02:08.036365 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:08.036302 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2" Oct 30 18:02:08.079046 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:08.078988 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2" podStartSLOduration=1.7830538009999999 podStartE2EDuration="2.078970074s" totalImagesPullingTime="295.916273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:02:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:02:08.077413175 +0000 UTC m=+2986.129513106" watchObservedRunningTime="2026-10-30 18:02:08.078970074 +0000 UTC m=+2986.131070005" Oct 30 18:02:08.416936 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:08.416849 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-6869fbf48b-jwbm4" podUID="c18e28aa-2d36-480c-9284-4b3b7212539e" containerName="payload-processing" containerID="cri-o://635a5110c3d6a4a9ceca52ba86dd5cac308e573e8def9fb019a861e1b10aa35c" gracePeriod=25 Oct 30 18:02:08.708156 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:08.708128 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-6869fbf48b-jwbm4" Oct 30 18:02:08.846892 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:08.846840 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c18e28aa-2d36-480c-9284-4b3b7212539e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c18e28aa-2d36-480c-9284-4b3b7212539e-plugins-config-volume\") pod \"c18e28aa-2d36-480c-9284-4b3b7212539e\" (UID: \"c18e28aa-2d36-480c-9284-4b3b7212539e\") " Oct 30 18:02:08.854233 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:08.854190 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c18e28aa-2d36-480c-9284-4b3b7212539e-kube-api-access-rxnn4\" (UniqueName: \"kubernetes.io/projected/c18e28aa-2d36-480c-9284-4b3b7212539e-kube-api-access-rxnn4\") pod \"c18e28aa-2d36-480c-9284-4b3b7212539e\" (UID: \"c18e28aa-2d36-480c-9284-4b3b7212539e\") " Oct 30 18:02:08.881895 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:08.867119 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c18e28aa-2d36-480c-9284-4b3b7212539e-plugins-config-volume" pod "c18e28aa-2d36-480c-9284-4b3b7212539e" (UID: "c18e28aa-2d36-480c-9284-4b3b7212539e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:02:08.902107 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:08.902045 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c18e28aa-2d36-480c-9284-4b3b7212539e-kube-api-access-rxnn4" pod "c18e28aa-2d36-480c-9284-4b3b7212539e" (UID: "c18e28aa-2d36-480c-9284-4b3b7212539e"). InnerVolumeSpecName "kube-api-access-rxnn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:02:08.957070 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:08.956992 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxnn4\" (UniqueName: \"kubernetes.io/projected/c18e28aa-2d36-480c-9284-4b3b7212539e-kube-api-access-rxnn4\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:02:08.957070 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:08.957018 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c18e28aa-2d36-480c-9284-4b3b7212539e-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:02:09.046002 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:09.045964 4965 generic.go:356] "Generic (PLEG): container finished" podID="c18e28aa-2d36-480c-9284-4b3b7212539e" containerID="635a5110c3d6a4a9ceca52ba86dd5cac308e573e8def9fb019a861e1b10aa35c" exitCode=0 Oct 30 18:02:09.046181 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:09.046014 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-6869fbf48b-jwbm4" Oct 30 18:02:09.046181 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:09.046022 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-6869fbf48b-jwbm4" event={"ID":"c18e28aa-2d36-480c-9284-4b3b7212539e","Type":"ContainerDied","Data":"635a5110c3d6a4a9ceca52ba86dd5cac308e573e8def9fb019a861e1b10aa35c"} Oct 30 18:02:09.046181 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:09.046073 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-584a12-6869fbf48b-jwbm4" event={"ID":"c18e28aa-2d36-480c-9284-4b3b7212539e","Type":"ContainerDied","Data":"d6c6c5eefd625d4a632edf24576a1a20562b08150cc46242b62de320552c59d3"} Oct 30 18:02:09.046181 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:09.046094 4965 scope.go:122] "RemoveContainer" containerID="635a5110c3d6a4a9ceca52ba86dd5cac308e573e8def9fb019a861e1b10aa35c" Oct 30 18:02:09.058179 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:09.058153 4965 scope.go:122] "RemoveContainer" containerID="635a5110c3d6a4a9ceca52ba86dd5cac308e573e8def9fb019a861e1b10aa35c" Oct 30 18:02:09.058477 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:02:09.058451 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"635a5110c3d6a4a9ceca52ba86dd5cac308e573e8def9fb019a861e1b10aa35c\": container with ID starting with 635a5110c3d6a4a9ceca52ba86dd5cac308e573e8def9fb019a861e1b10aa35c not found: ID does not exist" containerID="635a5110c3d6a4a9ceca52ba86dd5cac308e573e8def9fb019a861e1b10aa35c" Oct 30 18:02:09.058590 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:09.058482 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"635a5110c3d6a4a9ceca52ba86dd5cac308e573e8def9fb019a861e1b10aa35c"} err="failed to get container status \"635a5110c3d6a4a9ceca52ba86dd5cac308e573e8def9fb019a861e1b10aa35c\": rpc error: code = NotFound desc = could not find container \"635a5110c3d6a4a9ceca52ba86dd5cac308e573e8def9fb019a861e1b10aa35c\": container with ID starting with 635a5110c3d6a4a9ceca52ba86dd5cac308e573e8def9fb019a861e1b10aa35c not found: ID does not exist" Oct 30 18:02:09.107382 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:09.107359 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-584a12-6869fbf48b-jwbm4"] Oct 30 18:02:09.119813 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:09.119786 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-584a12-6869fbf48b-jwbm4"] Oct 30 18:02:09.936720 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:09.936688 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-5f4dcd6d56-kq9wj" Oct 30 18:02:09.937196 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:09.937123 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-7b54dd4f5-qtdnz" Oct 30 18:02:10.101855 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:10.101822 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c18e28aa-2d36-480c-9284-4b3b7212539e" path="/var/lib/kubelet/pods/c18e28aa-2d36-480c-9284-4b3b7212539e/volumes" Oct 30 18:02:14.053171 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:14.053142 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2" Oct 30 18:02:21.468018 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:21.467982 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-844694479f-vx68s"] Oct 30 18:02:22.826286 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:22.826254 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 18:02:22.833759 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:22.833735 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 18:02:26.468702 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:26.468640 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-844694479f-vx68s" podUID="e2e012d8-59ad-4aad-818d-b48aa6b35428" containerName="payload-pre-processing" containerID="cri-o://ee11c49174078f979f6e56b5b02adbcf5f47ca4f7919b9585ed7645f417c7aae" gracePeriod=25 Oct 30 18:02:26.757259 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:26.757229 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-844694479f-vx68s" Oct 30 18:02:26.880220 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:26.880188 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2e012d8-59ad-4aad-818d-b48aa6b35428-kube-api-access-w5wb6\" (UniqueName: \"kubernetes.io/projected/e2e012d8-59ad-4aad-818d-b48aa6b35428-kube-api-access-w5wb6\") pod \"e2e012d8-59ad-4aad-818d-b48aa6b35428\" (UID: \"e2e012d8-59ad-4aad-818d-b48aa6b35428\") " Oct 30 18:02:26.880562 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:26.880541 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2e012d8-59ad-4aad-818d-b48aa6b35428-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e012d8-59ad-4aad-818d-b48aa6b35428-plugins-config-volume\") pod \"e2e012d8-59ad-4aad-818d-b48aa6b35428\" (UID: \"e2e012d8-59ad-4aad-818d-b48aa6b35428\") " Oct 30 18:02:26.880949 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:26.880926 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2e012d8-59ad-4aad-818d-b48aa6b35428-plugins-config-volume" pod "e2e012d8-59ad-4aad-818d-b48aa6b35428" (UID: "e2e012d8-59ad-4aad-818d-b48aa6b35428"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:02:26.881142 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:26.881125 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e012d8-59ad-4aad-818d-b48aa6b35428-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:02:26.882372 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:26.882350 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2e012d8-59ad-4aad-818d-b48aa6b35428-kube-api-access-w5wb6" pod "e2e012d8-59ad-4aad-818d-b48aa6b35428" (UID: "e2e012d8-59ad-4aad-818d-b48aa6b35428"). InnerVolumeSpecName "kube-api-access-w5wb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:02:26.982961 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:26.982886 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5wb6\" (UniqueName: \"kubernetes.io/projected/e2e012d8-59ad-4aad-818d-b48aa6b35428-kube-api-access-w5wb6\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:02:27.240513 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:27.240434 4965 generic.go:356] "Generic (PLEG): container finished" podID="e2e012d8-59ad-4aad-818d-b48aa6b35428" containerID="ee11c49174078f979f6e56b5b02adbcf5f47ca4f7919b9585ed7645f417c7aae" exitCode=0 Oct 30 18:02:27.240513 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:27.240477 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-844694479f-vx68s" Oct 30 18:02:27.240705 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:27.240515 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-844694479f-vx68s" event={"ID":"e2e012d8-59ad-4aad-818d-b48aa6b35428","Type":"ContainerDied","Data":"ee11c49174078f979f6e56b5b02adbcf5f47ca4f7919b9585ed7645f417c7aae"} Oct 30 18:02:27.240705 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:27.240565 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-844694479f-vx68s" event={"ID":"e2e012d8-59ad-4aad-818d-b48aa6b35428","Type":"ContainerDied","Data":"bf40098e343598371468c30fd3789df5f11ab3c71fdfb5bc42f9c2cfb8144a8c"} Oct 30 18:02:27.240705 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:27.240590 4965 scope.go:122] "RemoveContainer" containerID="ee11c49174078f979f6e56b5b02adbcf5f47ca4f7919b9585ed7645f417c7aae" Oct 30 18:02:27.252323 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:27.252305 4965 scope.go:122] "RemoveContainer" containerID="ee11c49174078f979f6e56b5b02adbcf5f47ca4f7919b9585ed7645f417c7aae" Oct 30 18:02:27.252610 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:02:27.252583 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee11c49174078f979f6e56b5b02adbcf5f47ca4f7919b9585ed7645f417c7aae\": container with ID starting with ee11c49174078f979f6e56b5b02adbcf5f47ca4f7919b9585ed7645f417c7aae not found: ID does not exist" containerID="ee11c49174078f979f6e56b5b02adbcf5f47ca4f7919b9585ed7645f417c7aae" Oct 30 18:02:27.252702 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:27.252620 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee11c49174078f979f6e56b5b02adbcf5f47ca4f7919b9585ed7645f417c7aae"} err="failed to get container status \"ee11c49174078f979f6e56b5b02adbcf5f47ca4f7919b9585ed7645f417c7aae\": rpc error: code = NotFound desc = could not find container \"ee11c49174078f979f6e56b5b02adbcf5f47ca4f7919b9585ed7645f417c7aae\": container with ID starting with ee11c49174078f979f6e56b5b02adbcf5f47ca4f7919b9585ed7645f417c7aae not found: ID does not exist" Oct 30 18:02:27.296603 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:27.296576 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-844694479f-vx68s"] Oct 30 18:02:27.309977 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:27.309952 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-844694479f-vx68s"] Oct 30 18:02:28.100731 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.100702 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2e012d8-59ad-4aad-818d-b48aa6b35428" path="/var/lib/kubelet/pods/e2e012d8-59ad-4aad-818d-b48aa6b35428/volumes" Oct 30 18:02:28.462946 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.462851 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg"] Oct 30 18:02:28.464227 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.464200 4965 cpu_manager.go:414] "containerMap: removing container" podUID="c18e28aa-2d36-480c-9284-4b3b7212539e" containerName="payload-processing" Oct 30 18:02:28.464227 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.464222 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18e28aa-2d36-480c-9284-4b3b7212539e" containerName="payload-processing" Oct 30 18:02:28.464444 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.464261 4965 cpu_manager.go:414] "containerMap: removing container" podUID="e2e012d8-59ad-4aad-818d-b48aa6b35428" containerName="payload-pre-processing" Oct 30 18:02:28.464444 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.464266 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2e012d8-59ad-4aad-818d-b48aa6b35428" containerName="payload-pre-processing" Oct 30 18:02:28.464444 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.464378 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2e012d8-59ad-4aad-818d-b48aa6b35428" containerName="payload-pre-processing" Oct 30 18:02:28.464444 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.464396 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18e28aa-2d36-480c-9284-4b3b7212539e" containerName="payload-processing" Oct 30 18:02:28.476420 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.476395 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:28.498588 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.497718 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg"] Oct 30 18:02:28.504843 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.504811 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-95f55d-openshift-default-dockercfg-g6vqb\"" Oct 30 18:02:28.647886 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.640852 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a2a43ad-860d-465a-a29a-130c83307e87-istio-token\") pod \"e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:28.647886 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.640953 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-workload-socket\") pod \"e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:28.647886 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.641055 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79pp9\" (UniqueName: \"kubernetes.io/projected/4a2a43ad-860d-465a-a29a-130c83307e87-kube-api-access-79pp9\") pod \"e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:28.647886 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.641099 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a2a43ad-860d-465a-a29a-130c83307e87-istiod-ca-cert\") pod \"e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:28.647886 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.641144 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-istio-envoy\") pod \"e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:28.647886 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.641168 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a2a43ad-860d-465a-a29a-130c83307e87-istio-podinfo\") pod \"e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:28.647886 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.641261 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-credential-socket\") pod \"e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:28.647886 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.641334 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-workload-certs\") pod \"e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:28.647886 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.641490 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-istio-data\") pod \"e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:28.743120 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.743027 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-istio-data\") pod \"e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:28.743411 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.743388 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a2a43ad-860d-465a-a29a-130c83307e87-istio-token\") pod \"e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:28.743566 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.743547 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-workload-socket\") pod \"e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:28.743763 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.743745 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79pp9\" (UniqueName: \"kubernetes.io/projected/4a2a43ad-860d-465a-a29a-130c83307e87-kube-api-access-79pp9\") pod \"e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:28.745967 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.745936 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a2a43ad-860d-465a-a29a-130c83307e87-istiod-ca-cert\") pod \"e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:28.746163 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.744607 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-istio-data\") pod \"e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:28.746292 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.746273 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-istio-envoy\") pod \"e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:28.746406 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.746391 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a2a43ad-860d-465a-a29a-130c83307e87-istio-podinfo\") pod \"e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:28.746617 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.746597 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-credential-socket\") pod \"e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:28.746799 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.746780 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-workload-certs\") pod \"e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:28.747485 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.747460 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-workload-certs\") pod \"e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:28.769290 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.744836 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-workload-socket\") pod \"e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:28.769290 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.762785 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-credential-socket\") pod \"e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:28.769290 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.763173 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a2a43ad-860d-465a-a29a-130c83307e87-istiod-ca-cert\") pod \"e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:28.774256 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.774213 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-istio-envoy\") pod \"e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:28.779825 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.779754 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a2a43ad-860d-465a-a29a-130c83307e87-istio-podinfo\") pod \"e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:28.780372 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.780342 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a2a43ad-860d-465a-a29a-130c83307e87-istio-token\") pod \"e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:28.781136 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.781111 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79pp9\" (UniqueName: \"kubernetes.io/projected/4a2a43ad-860d-465a-a29a-130c83307e87-kube-api-access-79pp9\") pod \"e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:28.815473 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:28.808174 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:29.174514 ip-10-0-139-222 kubenswrapper[4965]: W1030 18:02:29.174467 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a2a43ad_860d_465a_a29a_130c83307e87.slice/crio-c4a1d8dc4bc01e9036c033e51c76eae7ac621ec9d9e177043e1896e6da787d10 WatchSource:0}: Error finding container c4a1d8dc4bc01e9036c033e51c76eae7ac621ec9d9e177043e1896e6da787d10: Status 404 returned error can't find the container with id c4a1d8dc4bc01e9036c033e51c76eae7ac621ec9d9e177043e1896e6da787d10 Oct 30 18:02:29.175379 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:29.175351 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg"] Oct 30 18:02:29.177475 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:29.177440 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 18:02:29.177568 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:29.177507 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 18:02:29.177568 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:29.177536 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 18:02:29.264765 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:29.264734 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" event={"ID":"4a2a43ad-860d-465a-a29a-130c83307e87","Type":"ContainerStarted","Data":"c4a1d8dc4bc01e9036c033e51c76eae7ac621ec9d9e177043e1896e6da787d10"} Oct 30 18:02:30.273312 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.273278 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" event={"ID":"4a2a43ad-860d-465a-a29a-130c83307e87","Type":"ContainerStarted","Data":"bca2e1151789163153b4d37b2637190d8e61196448ac81819f62c347e45ca03e"} Oct 30 18:02:30.331552 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.331498 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" podStartSLOduration=2.331485019 podStartE2EDuration="2.331485019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:02:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:02:30.32957797 +0000 UTC m=+3008.381677903" watchObservedRunningTime="2026-10-30 18:02:30.331485019 +0000 UTC m=+3008.383584949" Oct 30 18:02:30.373536 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.373502 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp"] Oct 30 18:02:30.378932 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.378908 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:30.394168 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.394140 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-d207f8-openshift-default-dockercfg-d2z4p\"" Oct 30 18:02:30.399689 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.399663 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp"] Oct 30 18:02:30.512467 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.511000 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306a7985-a3c4-46d3-a640-5663ace6b620-istio-podinfo\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:30.512467 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.511176 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-istio-data\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:30.512467 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.511209 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/306a7985-a3c4-46d3-a640-5663ace6b620-istio-token\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:30.512467 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.511233 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-workload-certs\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:30.512467 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.511260 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdmzd\" (UniqueName: \"kubernetes.io/projected/306a7985-a3c4-46d3-a640-5663ace6b620-kube-api-access-fdmzd\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:30.512467 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.511349 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-workload-socket\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:30.512467 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.511378 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-credential-socket\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:30.512467 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.511569 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306a7985-a3c4-46d3-a640-5663ace6b620-istiod-ca-cert\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:30.512467 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.511668 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-istio-envoy\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:30.613065 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.613027 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306a7985-a3c4-46d3-a640-5663ace6b620-istiod-ca-cert\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:30.613287 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.613133 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-istio-envoy\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:30.613287 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.613216 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306a7985-a3c4-46d3-a640-5663ace6b620-istio-podinfo\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:30.613419 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.613310 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-istio-data\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:30.613419 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.613339 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/306a7985-a3c4-46d3-a640-5663ace6b620-istio-token\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:30.613707 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.613645 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-istio-data\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:30.613787 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.613690 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-workload-certs\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:30.613787 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.613771 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdmzd\" (UniqueName: \"kubernetes.io/projected/306a7985-a3c4-46d3-a640-5663ace6b620-kube-api-access-fdmzd\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:30.613929 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.613838 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-workload-socket\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:30.613929 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.613884 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-credential-socket\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:30.613929 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.613889 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306a7985-a3c4-46d3-a640-5663ace6b620-istiod-ca-cert\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:30.614097 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.613927 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-workload-certs\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:30.614160 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.614134 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-workload-socket\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:30.614833 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.614812 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-credential-socket\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:30.615673 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.615646 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-istio-envoy\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:30.616201 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.616184 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306a7985-a3c4-46d3-a640-5663ace6b620-istio-podinfo\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:30.636606 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.636580 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/306a7985-a3c4-46d3-a640-5663ace6b620-istio-token\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:30.636927 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.636908 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdmzd\" (UniqueName: \"kubernetes.io/projected/306a7985-a3c4-46d3-a640-5663ace6b620-kube-api-access-fdmzd\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:30.694548 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.694517 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:30.812185 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.811991 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:30.819426 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:30.819396 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:31.045719 ip-10-0-139-222 kubenswrapper[4965]: W1030 18:02:31.045682 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod306a7985_a3c4_46d3_a640_5663ace6b620.slice/crio-5010cb16392103ac2864b952bb4e79e964988f69594f70c3a608513059bd5264 WatchSource:0}: Error finding container 5010cb16392103ac2864b952bb4e79e964988f69594f70c3a608513059bd5264: Status 404 returned error can't find the container with id 5010cb16392103ac2864b952bb4e79e964988f69594f70c3a608513059bd5264 Oct 30 18:02:31.046597 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:31.046572 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp"] Oct 30 18:02:31.049065 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:31.049026 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 18:02:31.049173 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:31.049112 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 18:02:31.049173 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:31.049160 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 18:02:31.282398 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:31.282360 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" event={"ID":"306a7985-a3c4-46d3-a640-5663ace6b620","Type":"ContainerStarted","Data":"0b5949d4be3e560ebe13e9a3048c66b5b94c5d558e2da9b444c02a9e0c5a7a41"} Oct 30 18:02:31.282912 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:31.282421 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" event={"ID":"306a7985-a3c4-46d3-a640-5663ace6b620","Type":"ContainerStarted","Data":"5010cb16392103ac2864b952bb4e79e964988f69594f70c3a608513059bd5264"} Oct 30 18:02:31.282912 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:31.282638 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:31.283583 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:31.283561 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:02:31.336498 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:31.336441 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" podStartSLOduration=1.336422995 podStartE2EDuration="1.336422995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:02:31.33041522 +0000 UTC m=+3009.382515149" watchObservedRunningTime="2026-10-30 18:02:31.336422995 +0000 UTC m=+3009.388522927" Oct 30 18:02:31.694929 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:31.694823 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:32.697348 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:32.697313 4965 patch_prober.go:29] interesting pod/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 18:02:32.697807 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:32.697373 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" podUID="306a7985-a3c4-46d3-a640-5663ace6b620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 18:02:33.695552 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:33.695517 4965 patch_prober.go:29] interesting pod/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 18:02:33.695737 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:33.695603 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" podUID="306a7985-a3c4-46d3-a640-5663ace6b620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 18:02:34.721883 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:34.721211 4965 patch_prober.go:29] interesting pod/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 18:02:34.721883 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:34.721300 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" podUID="306a7985-a3c4-46d3-a640-5663ace6b620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 18:02:35.695644 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:35.695607 4965 patch_prober.go:29] interesting pod/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 18:02:35.695831 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:35.695671 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" podUID="306a7985-a3c4-46d3-a640-5663ace6b620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 18:02:36.013988 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:02:36.013903 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:02:36.694835 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:36.694795 4965 patch_prober.go:29] interesting pod/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 18:02:36.695069 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:36.694849 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" podUID="306a7985-a3c4-46d3-a640-5663ace6b620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 18:02:37.695381 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:37.695349 4965 patch_prober.go:29] interesting pod/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 18:02:37.695749 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:37.695412 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" podUID="306a7985-a3c4-46d3-a640-5663ace6b620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 18:02:38.700402 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:38.700372 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:38.700892 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:38.700847 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:38.701458 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:38.701437 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:02:39.817363 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:02:39.817324 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:03:05.998127 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:03:05.998092 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:03:07.377992 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:07.377956 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-bc6c5656c-mp82n"] Oct 30 18:03:07.378505 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:07.378165 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-bc6c5656c-mp82n" podUID="fa4299c9-7f64-4e69-8f31-0499a36a4179" containerName="payload-pre-processing" containerID="cri-o://e7686aef5d7aa5530a25d6817eae28a89abba77a697befece07553ea67abcc4d" gracePeriod=30 Oct 30 18:03:07.699886 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:07.689333 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-bc6c5656c-mp82n" Oct 30 18:03:07.889200 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:07.888183 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa4299c9-7f64-4e69-8f31-0499a36a4179-kube-api-access-nvtgj\" (UniqueName: \"kubernetes.io/projected/fa4299c9-7f64-4e69-8f31-0499a36a4179-kube-api-access-nvtgj\") pod \"fa4299c9-7f64-4e69-8f31-0499a36a4179\" (UID: \"fa4299c9-7f64-4e69-8f31-0499a36a4179\") " Oct 30 18:03:07.889200 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:07.888489 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa4299c9-7f64-4e69-8f31-0499a36a4179-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4299c9-7f64-4e69-8f31-0499a36a4179-plugins-config-volume\") pod \"fa4299c9-7f64-4e69-8f31-0499a36a4179\" (UID: \"fa4299c9-7f64-4e69-8f31-0499a36a4179\") " Oct 30 18:03:07.890116 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:07.890068 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa4299c9-7f64-4e69-8f31-0499a36a4179-plugins-config-volume" pod "fa4299c9-7f64-4e69-8f31-0499a36a4179" (UID: "fa4299c9-7f64-4e69-8f31-0499a36a4179"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:03:07.893051 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:07.892823 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa4299c9-7f64-4e69-8f31-0499a36a4179-kube-api-access-nvtgj" pod "fa4299c9-7f64-4e69-8f31-0499a36a4179" (UID: "fa4299c9-7f64-4e69-8f31-0499a36a4179"). InnerVolumeSpecName "kube-api-access-nvtgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:07.896910 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:07.894328 4965 generic.go:356] "Generic (PLEG): container finished" podID="fa4299c9-7f64-4e69-8f31-0499a36a4179" containerID="e7686aef5d7aa5530a25d6817eae28a89abba77a697befece07553ea67abcc4d" exitCode=0 Oct 30 18:03:07.896910 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:07.894378 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-bc6c5656c-mp82n" Oct 30 18:03:07.896910 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:07.894541 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-bc6c5656c-mp82n" event={"ID":"fa4299c9-7f64-4e69-8f31-0499a36a4179","Type":"ContainerDied","Data":"e7686aef5d7aa5530a25d6817eae28a89abba77a697befece07553ea67abcc4d"} Oct 30 18:03:07.896910 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:07.894582 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-bc6c5656c-mp82n" event={"ID":"fa4299c9-7f64-4e69-8f31-0499a36a4179","Type":"ContainerDied","Data":"a538cd19513c18a7c9e57bd55c5e65ec6779ca751190c89c7d4d2bf29216efb2"} Oct 30 18:03:07.896910 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:07.894608 4965 scope.go:122] "RemoveContainer" containerID="e7686aef5d7aa5530a25d6817eae28a89abba77a697befece07553ea67abcc4d" Oct 30 18:03:07.941109 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:07.940122 4965 scope.go:122] "RemoveContainer" containerID="e7686aef5d7aa5530a25d6817eae28a89abba77a697befece07553ea67abcc4d" Oct 30 18:03:07.941109 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:03:07.940440 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7686aef5d7aa5530a25d6817eae28a89abba77a697befece07553ea67abcc4d\": container with ID starting with e7686aef5d7aa5530a25d6817eae28a89abba77a697befece07553ea67abcc4d not found: ID does not exist" containerID="e7686aef5d7aa5530a25d6817eae28a89abba77a697befece07553ea67abcc4d" Oct 30 18:03:07.941109 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:07.940468 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7686aef5d7aa5530a25d6817eae28a89abba77a697befece07553ea67abcc4d"} err="failed to get container status \"e7686aef5d7aa5530a25d6817eae28a89abba77a697befece07553ea67abcc4d\": rpc error: code = NotFound desc = could not find container \"e7686aef5d7aa5530a25d6817eae28a89abba77a697befece07553ea67abcc4d\": container with ID starting with e7686aef5d7aa5530a25d6817eae28a89abba77a697befece07553ea67abcc4d not found: ID does not exist" Oct 30 18:03:07.983531 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:07.983502 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-bc6c5656c-mp82n"] Oct 30 18:03:07.989533 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:07.989505 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-21870f-bc6c5656c-mp82n"] Oct 30 18:03:07.990443 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:07.990420 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4299c9-7f64-4e69-8f31-0499a36a4179-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:03:07.990534 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:07.990444 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvtgj\" (UniqueName: \"kubernetes.io/projected/fa4299c9-7f64-4e69-8f31-0499a36a4179-kube-api-access-nvtgj\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:03:08.101456 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:08.101422 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa4299c9-7f64-4e69-8f31-0499a36a4179" path="/var/lib/kubelet/pods/fa4299c9-7f64-4e69-8f31-0499a36a4179/volumes" Oct 30 18:03:10.542702 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:10.542667 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-7458959b4c-fxzzm"] Oct 30 18:03:10.544631 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:10.544599 4965 cpu_manager.go:414] "containerMap: removing container" podUID="fa4299c9-7f64-4e69-8f31-0499a36a4179" containerName="payload-pre-processing" Oct 30 18:03:10.544844 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:10.544826 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa4299c9-7f64-4e69-8f31-0499a36a4179" containerName="payload-pre-processing" Oct 30 18:03:10.545496 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:10.545238 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa4299c9-7f64-4e69-8f31-0499a36a4179" containerName="payload-pre-processing" Oct 30 18:03:10.551937 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:10.551910 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-7458959b4c-fxzzm" Oct 30 18:03:10.553998 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:10.553973 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-7458959b4c-fxzzm"] Oct 30 18:03:10.566829 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:10.566802 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d022f31a-8a4f-4a1c-9b99-2b53227e9d49-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b5cbb4-7458959b4c-fxzzm\" (UID: \"d022f31a-8a4f-4a1c-9b99-2b53227e9d49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-7458959b4c-fxzzm" Oct 30 18:03:10.567055 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:10.567033 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsb5w\" (UniqueName: \"kubernetes.io/projected/d022f31a-8a4f-4a1c-9b99-2b53227e9d49-kube-api-access-xsb5w\") pod \"payload-pre-processing-e2e-ipp-b-b5cbb4-7458959b4c-fxzzm\" (UID: \"d022f31a-8a4f-4a1c-9b99-2b53227e9d49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-7458959b4c-fxzzm" Oct 30 18:03:10.567131 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:10.567092 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b5cbb4\"" Oct 30 18:03:10.567403 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:10.567383 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b5cbb4-dockercfg-4nf7q\"" Oct 30 18:03:10.668446 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:10.668411 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d022f31a-8a4f-4a1c-9b99-2b53227e9d49-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b5cbb4-7458959b4c-fxzzm\" (UID: \"d022f31a-8a4f-4a1c-9b99-2b53227e9d49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-7458959b4c-fxzzm" Oct 30 18:03:10.668633 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:10.668566 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsb5w\" (UniqueName: \"kubernetes.io/projected/d022f31a-8a4f-4a1c-9b99-2b53227e9d49-kube-api-access-xsb5w\") pod \"payload-pre-processing-e2e-ipp-b-b5cbb4-7458959b4c-fxzzm\" (UID: \"d022f31a-8a4f-4a1c-9b99-2b53227e9d49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-7458959b4c-fxzzm" Oct 30 18:03:10.669139 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:10.669115 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d022f31a-8a4f-4a1c-9b99-2b53227e9d49-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b5cbb4-7458959b4c-fxzzm\" (UID: \"d022f31a-8a4f-4a1c-9b99-2b53227e9d49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-7458959b4c-fxzzm" Oct 30 18:03:10.692344 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:10.692317 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsb5w\" (UniqueName: \"kubernetes.io/projected/d022f31a-8a4f-4a1c-9b99-2b53227e9d49-kube-api-access-xsb5w\") pod \"payload-pre-processing-e2e-ipp-b-b5cbb4-7458959b4c-fxzzm\" (UID: \"d022f31a-8a4f-4a1c-9b99-2b53227e9d49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-7458959b4c-fxzzm" Oct 30 18:03:10.872025 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:10.871990 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-7458959b4c-fxzzm" Oct 30 18:03:11.187093 ip-10-0-139-222 kubenswrapper[4965]: W1030 18:03:11.187061 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd022f31a_8a4f_4a1c_9b99_2b53227e9d49.slice/crio-b804d16ddeca58a6f67d322de20930a38a1c9b0bb0b95e6c3e6ac7c40bd633b9 WatchSource:0}: Error finding container b804d16ddeca58a6f67d322de20930a38a1c9b0bb0b95e6c3e6ac7c40bd633b9: Status 404 returned error can't find the container with id b804d16ddeca58a6f67d322de20930a38a1c9b0bb0b95e6c3e6ac7c40bd633b9 Oct 30 18:03:11.187346 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:11.187327 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-7458959b4c-fxzzm"] Oct 30 18:03:11.974832 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:11.974782 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-7458959b4c-fxzzm" event={"ID":"d022f31a-8a4f-4a1c-9b99-2b53227e9d49","Type":"ContainerStarted","Data":"5c9c2ca0519c46722b05a346c30c18561e5210e37ce345fd51796e939bd94578"} Oct 30 18:03:11.975360 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:11.974857 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-7458959b4c-fxzzm" event={"ID":"d022f31a-8a4f-4a1c-9b99-2b53227e9d49","Type":"ContainerStarted","Data":"b804d16ddeca58a6f67d322de20930a38a1c9b0bb0b95e6c3e6ac7c40bd633b9"} Oct 30 18:03:11.975360 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:11.974913 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-7458959b4c-fxzzm" Oct 30 18:03:12.034602 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:12.034540 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-7458959b4c-fxzzm" podStartSLOduration=2.034520652 podStartE2EDuration="2.034520652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:12.031617426 +0000 UTC m=+3050.083717368" watchObservedRunningTime="2026-10-30 18:03:12.034520652 +0000 UTC m=+3050.086620582" Oct 30 18:03:22.987036 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:22.987006 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-7458959b4c-fxzzm" Oct 30 18:03:29.214313 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:29.214275 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-90b48a-7d6d7f978b-v22m7"] Oct 30 18:03:29.245642 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:29.245613 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-90b48a-745c46d66c-v42dh"] Oct 30 18:03:34.214753 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:34.214708 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-90b48a-7d6d7f978b-v22m7" podUID="f4296e00-981c-4a22-8f06-d8fc997c359e" containerName="payload-processing" containerID="cri-o://5a5e9c989bb0b699c9086b3feec00d0823c66778c5c94fd30614142db2a2ca7e" gracePeriod=25 Oct 30 18:03:34.246785 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:34.246742 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-90b48a-745c46d66c-v42dh" podUID="844e6c1f-9e6d-4890-93e0-4216c37bc999" containerName="payload-pre-processing" containerID="cri-o://ca15ce206f4bd06d526942ad08dd24bcbda6296f023787645ef8cdb0dfd676c4" gracePeriod=25 Oct 30 18:03:35.921891 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:35.921080 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:03:35.997666 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:03:35.997632 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:03:43.491966 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:43.491930 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-90b48a-745c46d66c-v42dh" podUID="844e6c1f-9e6d-4890-93e0-4216c37bc999" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:03:43.492462 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:43.491935 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-90b48a-7d6d7f978b-v22m7" podUID="f4296e00-981c-4a22-8f06-d8fc997c359e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:03:47.442517 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:47.442481 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e927d7-5f4dcd6d56-kq9wj"] Oct 30 18:03:47.442957 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:47.442734 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-5f4dcd6d56-kq9wj" podUID="e84b153b-396c-4135-bd60-a92cac969337" containerName="payload-processing" containerID="cri-o://0ec8f0553e377e9102eab0ebd23badf478704ba0eac0fcf9e390c09980dce1dc" gracePeriod=30 Oct 30 18:03:47.477518 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:47.477473 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-7b54dd4f5-qtdnz"] Oct 30 18:03:47.478027 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:47.477996 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-7b54dd4f5-qtdnz" podUID="a23856b6-65d8-489d-9bae-299672aa44d2" containerName="payload-pre-processing" containerID="cri-o://aed30f9ab6f0961ea9f58bec2c5b2ce775a05620cb36758ded6a344d292f4d19" gracePeriod=30 Oct 30 18:03:47.889519 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:47.889170 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-5f4dcd6d56-kq9wj" Oct 30 18:03:48.015575 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:48.015314 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-7b54dd4f5-qtdnz" Oct 30 18:03:48.078605 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:48.078547 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e84b153b-396c-4135-bd60-a92cac969337-kube-api-access-6rqr7\" (UniqueName: \"kubernetes.io/projected/e84b153b-396c-4135-bd60-a92cac969337-kube-api-access-6rqr7\") pod \"e84b153b-396c-4135-bd60-a92cac969337\" (UID: \"e84b153b-396c-4135-bd60-a92cac969337\") " Oct 30 18:03:48.078805 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:48.078690 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e84b153b-396c-4135-bd60-a92cac969337-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e84b153b-396c-4135-bd60-a92cac969337-plugins-config-volume\") pod \"e84b153b-396c-4135-bd60-a92cac969337\" (UID: \"e84b153b-396c-4135-bd60-a92cac969337\") " Oct 30 18:03:48.079371 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:48.079344 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e84b153b-396c-4135-bd60-a92cac969337-plugins-config-volume" pod "e84b153b-396c-4135-bd60-a92cac969337" (UID: "e84b153b-396c-4135-bd60-a92cac969337"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:03:48.081152 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:48.081126 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e84b153b-396c-4135-bd60-a92cac969337-kube-api-access-6rqr7" pod "e84b153b-396c-4135-bd60-a92cac969337" (UID: "e84b153b-396c-4135-bd60-a92cac969337"). InnerVolumeSpecName "kube-api-access-6rqr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:48.183883 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:48.183669 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a23856b6-65d8-489d-9bae-299672aa44d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a23856b6-65d8-489d-9bae-299672aa44d2-plugins-config-volume\") pod \"a23856b6-65d8-489d-9bae-299672aa44d2\" (UID: \"a23856b6-65d8-489d-9bae-299672aa44d2\") " Oct 30 18:03:48.183883 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:48.183736 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a23856b6-65d8-489d-9bae-299672aa44d2-kube-api-access-vtlds\" (UniqueName: \"kubernetes.io/projected/a23856b6-65d8-489d-9bae-299672aa44d2-kube-api-access-vtlds\") pod \"a23856b6-65d8-489d-9bae-299672aa44d2\" (UID: \"a23856b6-65d8-489d-9bae-299672aa44d2\") " Oct 30 18:03:48.187885 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:48.184536 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e84b153b-396c-4135-bd60-a92cac969337-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:03:48.187885 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:48.184566 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rqr7\" (UniqueName: \"kubernetes.io/projected/e84b153b-396c-4135-bd60-a92cac969337-kube-api-access-6rqr7\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:03:48.187885 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:48.185508 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a23856b6-65d8-489d-9bae-299672aa44d2-plugins-config-volume" pod "a23856b6-65d8-489d-9bae-299672aa44d2" (UID: "a23856b6-65d8-489d-9bae-299672aa44d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:03:48.198515 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:48.197457 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23856b6-65d8-489d-9bae-299672aa44d2-kube-api-access-vtlds" pod "a23856b6-65d8-489d-9bae-299672aa44d2" (UID: "a23856b6-65d8-489d-9bae-299672aa44d2"). InnerVolumeSpecName "kube-api-access-vtlds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:48.286391 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:48.286311 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a23856b6-65d8-489d-9bae-299672aa44d2-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:03:48.286391 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:48.286349 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtlds\" (UniqueName: \"kubernetes.io/projected/a23856b6-65d8-489d-9bae-299672aa44d2-kube-api-access-vtlds\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:03:48.332822 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:48.332787 4965 generic.go:356] "Generic (PLEG): container finished" podID="a23856b6-65d8-489d-9bae-299672aa44d2" containerID="aed30f9ab6f0961ea9f58bec2c5b2ce775a05620cb36758ded6a344d292f4d19" exitCode=0 Oct 30 18:03:48.333031 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:48.332837 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-7b54dd4f5-qtdnz" Oct 30 18:03:48.333031 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:48.332907 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-7b54dd4f5-qtdnz" event={"ID":"a23856b6-65d8-489d-9bae-299672aa44d2","Type":"ContainerDied","Data":"aed30f9ab6f0961ea9f58bec2c5b2ce775a05620cb36758ded6a344d292f4d19"} Oct 30 18:03:48.333031 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:48.332973 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-7b54dd4f5-qtdnz" event={"ID":"a23856b6-65d8-489d-9bae-299672aa44d2","Type":"ContainerDied","Data":"350153c5f6e928f440c41347b0e65c25435eeb2909f9c3b908bd9761f9ea9d53"} Oct 30 18:03:48.333031 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:48.333002 4965 scope.go:122] "RemoveContainer" containerID="aed30f9ab6f0961ea9f58bec2c5b2ce775a05620cb36758ded6a344d292f4d19" Oct 30 18:03:48.334447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:48.334427 4965 generic.go:356] "Generic (PLEG): container finished" podID="e84b153b-396c-4135-bd60-a92cac969337" containerID="0ec8f0553e377e9102eab0ebd23badf478704ba0eac0fcf9e390c09980dce1dc" exitCode=0 Oct 30 18:03:48.334593 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:48.334577 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-5f4dcd6d56-kq9wj" event={"ID":"e84b153b-396c-4135-bd60-a92cac969337","Type":"ContainerDied","Data":"0ec8f0553e377e9102eab0ebd23badf478704ba0eac0fcf9e390c09980dce1dc"} Oct 30 18:03:48.334669 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:48.334621 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-5f4dcd6d56-kq9wj" event={"ID":"e84b153b-396c-4135-bd60-a92cac969337","Type":"ContainerDied","Data":"b80591768fc39715c2e4c4af3c1a616083a3f83f40d9f9cbda79123f01f580b1"} Oct 30 18:03:48.334735 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:48.334685 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e927d7-5f4dcd6d56-kq9wj" Oct 30 18:03:48.344814 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:48.344798 4965 scope.go:122] "RemoveContainer" containerID="aed30f9ab6f0961ea9f58bec2c5b2ce775a05620cb36758ded6a344d292f4d19" Oct 30 18:03:48.345111 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:03:48.345092 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aed30f9ab6f0961ea9f58bec2c5b2ce775a05620cb36758ded6a344d292f4d19\": container with ID starting with aed30f9ab6f0961ea9f58bec2c5b2ce775a05620cb36758ded6a344d292f4d19 not found: ID does not exist" containerID="aed30f9ab6f0961ea9f58bec2c5b2ce775a05620cb36758ded6a344d292f4d19" Oct 30 18:03:48.345178 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:48.345120 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aed30f9ab6f0961ea9f58bec2c5b2ce775a05620cb36758ded6a344d292f4d19"} err="failed to get container status \"aed30f9ab6f0961ea9f58bec2c5b2ce775a05620cb36758ded6a344d292f4d19\": rpc error: code = NotFound desc = could not find container \"aed30f9ab6f0961ea9f58bec2c5b2ce775a05620cb36758ded6a344d292f4d19\": container with ID starting with aed30f9ab6f0961ea9f58bec2c5b2ce775a05620cb36758ded6a344d292f4d19 not found: ID does not exist" Oct 30 18:03:48.345178 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:48.345137 4965 scope.go:122] "RemoveContainer" containerID="0ec8f0553e377e9102eab0ebd23badf478704ba0eac0fcf9e390c09980dce1dc" Oct 30 18:03:48.355559 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:48.355542 4965 scope.go:122] "RemoveContainer" containerID="0ec8f0553e377e9102eab0ebd23badf478704ba0eac0fcf9e390c09980dce1dc" Oct 30 18:03:48.355826 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:03:48.355807 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ec8f0553e377e9102eab0ebd23badf478704ba0eac0fcf9e390c09980dce1dc\": container with ID starting with 0ec8f0553e377e9102eab0ebd23badf478704ba0eac0fcf9e390c09980dce1dc not found: ID does not exist" containerID="0ec8f0553e377e9102eab0ebd23badf478704ba0eac0fcf9e390c09980dce1dc" Oct 30 18:03:48.355917 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:48.355834 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec8f0553e377e9102eab0ebd23badf478704ba0eac0fcf9e390c09980dce1dc"} err="failed to get container status \"0ec8f0553e377e9102eab0ebd23badf478704ba0eac0fcf9e390c09980dce1dc\": rpc error: code = NotFound desc = could not find container \"0ec8f0553e377e9102eab0ebd23badf478704ba0eac0fcf9e390c09980dce1dc\": container with ID starting with 0ec8f0553e377e9102eab0ebd23badf478704ba0eac0fcf9e390c09980dce1dc not found: ID does not exist" Oct 30 18:03:48.388373 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:48.388341 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-7b54dd4f5-qtdnz"] Oct 30 18:03:48.403958 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:48.403932 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e927d7-7b54dd4f5-qtdnz"] Oct 30 18:03:48.426388 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:48.426365 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e927d7-5f4dcd6d56-kq9wj"] Oct 30 18:03:48.430278 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:48.430256 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e927d7-5f4dcd6d56-kq9wj"] Oct 30 18:03:50.101837 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:50.101803 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a23856b6-65d8-489d-9bae-299672aa44d2" path="/var/lib/kubelet/pods/a23856b6-65d8-489d-9bae-299672aa44d2/volumes" Oct 30 18:03:50.102349 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:50.102314 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e84b153b-396c-4135-bd60-a92cac969337" path="/var/lib/kubelet/pods/e84b153b-396c-4135-bd60-a92cac969337/volumes" Oct 30 18:03:51.708354 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:51.708320 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg"] Oct 30 18:03:51.708855 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:51.708538 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" podUID="6e3c3d72-4ad1-4368-93e4-5f429a7f80e9" containerName="istio-proxy" containerID="cri-o://d3eb8d14fd0c7a2c882f5077197e73d9111a4619bbf7aa9152dc83749d177814" gracePeriod=30 Oct 30 18:03:53.493709 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:53.493671 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-90b48a-745c46d66c-v42dh" podUID="844e6c1f-9e6d-4890-93e0-4216c37bc999" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:03:53.494130 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:53.493758 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-90b48a-7d6d7f978b-v22m7" podUID="f4296e00-981c-4a22-8f06-d8fc997c359e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:03:54.415553 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:54.415166 4965 generic.go:356] "Generic (PLEG): container finished" podID="f4296e00-981c-4a22-8f06-d8fc997c359e" containerID="5a5e9c989bb0b699c9086b3feec00d0823c66778c5c94fd30614142db2a2ca7e" exitCode=0 Oct 30 18:03:54.415553 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:54.415469 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-90b48a-7d6d7f978b-v22m7" event={"ID":"f4296e00-981c-4a22-8f06-d8fc997c359e","Type":"ContainerDied","Data":"5a5e9c989bb0b699c9086b3feec00d0823c66778c5c94fd30614142db2a2ca7e"} Oct 30 18:03:54.704958 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:54.704932 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-90b48a-7d6d7f978b-v22m7" Oct 30 18:03:54.708893 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:54.708854 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-90b48a-745c46d66c-v42dh" Oct 30 18:03:54.803672 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:54.803591 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/844e6c1f-9e6d-4890-93e0-4216c37bc999-kube-api-access-7mtpw\" (UniqueName: \"kubernetes.io/projected/844e6c1f-9e6d-4890-93e0-4216c37bc999-kube-api-access-7mtpw\") pod \"844e6c1f-9e6d-4890-93e0-4216c37bc999\" (UID: \"844e6c1f-9e6d-4890-93e0-4216c37bc999\") " Oct 30 18:03:54.803873 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:54.803680 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4296e00-981c-4a22-8f06-d8fc997c359e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4296e00-981c-4a22-8f06-d8fc997c359e-plugins-config-volume\") pod \"f4296e00-981c-4a22-8f06-d8fc997c359e\" (UID: \"f4296e00-981c-4a22-8f06-d8fc997c359e\") " Oct 30 18:03:54.803873 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:54.803816 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4296e00-981c-4a22-8f06-d8fc997c359e-kube-api-access-xxpq5\" (UniqueName: \"kubernetes.io/projected/f4296e00-981c-4a22-8f06-d8fc997c359e-kube-api-access-xxpq5\") pod \"f4296e00-981c-4a22-8f06-d8fc997c359e\" (UID: \"f4296e00-981c-4a22-8f06-d8fc997c359e\") " Oct 30 18:03:54.804021 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:54.803973 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/844e6c1f-9e6d-4890-93e0-4216c37bc999-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844e6c1f-9e6d-4890-93e0-4216c37bc999-plugins-config-volume\") pod \"844e6c1f-9e6d-4890-93e0-4216c37bc999\" (UID: \"844e6c1f-9e6d-4890-93e0-4216c37bc999\") " Oct 30 18:03:54.804151 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:54.804124 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4296e00-981c-4a22-8f06-d8fc997c359e-plugins-config-volume" pod "f4296e00-981c-4a22-8f06-d8fc997c359e" (UID: "f4296e00-981c-4a22-8f06-d8fc997c359e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:03:54.804394 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:54.804371 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/844e6c1f-9e6d-4890-93e0-4216c37bc999-plugins-config-volume" pod "844e6c1f-9e6d-4890-93e0-4216c37bc999" (UID: "844e6c1f-9e6d-4890-93e0-4216c37bc999"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:03:54.804807 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:54.804779 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844e6c1f-9e6d-4890-93e0-4216c37bc999-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:03:54.804807 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:54.804808 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4296e00-981c-4a22-8f06-d8fc997c359e-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:03:54.806226 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:54.806198 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/844e6c1f-9e6d-4890-93e0-4216c37bc999-kube-api-access-7mtpw" pod "844e6c1f-9e6d-4890-93e0-4216c37bc999" (UID: "844e6c1f-9e6d-4890-93e0-4216c37bc999"). InnerVolumeSpecName "kube-api-access-7mtpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:54.806354 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:54.806261 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4296e00-981c-4a22-8f06-d8fc997c359e-kube-api-access-xxpq5" pod "f4296e00-981c-4a22-8f06-d8fc997c359e" (UID: "f4296e00-981c-4a22-8f06-d8fc997c359e"). InnerVolumeSpecName "kube-api-access-xxpq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:54.906343 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:54.906306 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mtpw\" (UniqueName: \"kubernetes.io/projected/844e6c1f-9e6d-4890-93e0-4216c37bc999-kube-api-access-7mtpw\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:03:54.906343 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:54.906336 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxpq5\" (UniqueName: \"kubernetes.io/projected/f4296e00-981c-4a22-8f06-d8fc997c359e-kube-api-access-xxpq5\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:03:55.425020 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:55.424988 4965 generic.go:356] "Generic (PLEG): container finished" podID="844e6c1f-9e6d-4890-93e0-4216c37bc999" containerID="ca15ce206f4bd06d526942ad08dd24bcbda6296f023787645ef8cdb0dfd676c4" exitCode=0 Oct 30 18:03:55.425232 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:55.425032 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-90b48a-745c46d66c-v42dh" Oct 30 18:03:55.425232 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:55.425066 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-90b48a-745c46d66c-v42dh" event={"ID":"844e6c1f-9e6d-4890-93e0-4216c37bc999","Type":"ContainerDied","Data":"ca15ce206f4bd06d526942ad08dd24bcbda6296f023787645ef8cdb0dfd676c4"} Oct 30 18:03:55.425232 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:55.425125 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-90b48a-745c46d66c-v42dh" event={"ID":"844e6c1f-9e6d-4890-93e0-4216c37bc999","Type":"ContainerDied","Data":"9396e9ebd4663fb2e3f56798efa1419ffdc0c519fa14ce421f23d6d536cdb81a"} Oct 30 18:03:55.425232 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:55.425157 4965 scope.go:122] "RemoveContainer" containerID="ca15ce206f4bd06d526942ad08dd24bcbda6296f023787645ef8cdb0dfd676c4" Oct 30 18:03:55.426776 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:55.426761 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-90b48a-7d6d7f978b-v22m7" Oct 30 18:03:55.426906 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:55.426769 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-90b48a-7d6d7f978b-v22m7" event={"ID":"f4296e00-981c-4a22-8f06-d8fc997c359e","Type":"ContainerDied","Data":"48ac9ef20b3bb43dd88a433b6833330d498b2fb6f71e963a61cd884e370d47df"} Oct 30 18:03:55.438709 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:55.438690 4965 scope.go:122] "RemoveContainer" containerID="ca15ce206f4bd06d526942ad08dd24bcbda6296f023787645ef8cdb0dfd676c4" Oct 30 18:03:55.439096 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:03:55.439070 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca15ce206f4bd06d526942ad08dd24bcbda6296f023787645ef8cdb0dfd676c4\": container with ID starting with ca15ce206f4bd06d526942ad08dd24bcbda6296f023787645ef8cdb0dfd676c4 not found: ID does not exist" containerID="ca15ce206f4bd06d526942ad08dd24bcbda6296f023787645ef8cdb0dfd676c4" Oct 30 18:03:55.439171 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:55.439107 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca15ce206f4bd06d526942ad08dd24bcbda6296f023787645ef8cdb0dfd676c4"} err="failed to get container status \"ca15ce206f4bd06d526942ad08dd24bcbda6296f023787645ef8cdb0dfd676c4\": rpc error: code = NotFound desc = could not find container \"ca15ce206f4bd06d526942ad08dd24bcbda6296f023787645ef8cdb0dfd676c4\": container with ID starting with ca15ce206f4bd06d526942ad08dd24bcbda6296f023787645ef8cdb0dfd676c4 not found: ID does not exist" Oct 30 18:03:55.439171 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:55.439131 4965 scope.go:122] "RemoveContainer" containerID="5a5e9c989bb0b699c9086b3feec00d0823c66778c5c94fd30614142db2a2ca7e" Oct 30 18:03:55.482446 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:55.482411 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-90b48a-7d6d7f978b-v22m7"] Oct 30 18:03:55.496176 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:55.496082 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-90b48a-7d6d7f978b-v22m7"] Oct 30 18:03:55.535546 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:55.535512 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-90b48a-745c46d66c-v42dh"] Oct 30 18:03:55.540421 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:55.540396 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-90b48a-745c46d66c-v42dh"] Oct 30 18:03:56.100501 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:56.100469 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="844e6c1f-9e6d-4890-93e0-4216c37bc999" path="/var/lib/kubelet/pods/844e6c1f-9e6d-4890-93e0-4216c37bc999/volumes" Oct 30 18:03:56.100957 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:56.100942 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4296e00-981c-4a22-8f06-d8fc997c359e" path="/var/lib/kubelet/pods/f4296e00-981c-4a22-8f06-d8fc997c359e/volumes" Oct 30 18:03:56.994630 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:56.994569 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 18:03:57.055495 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.055454 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istio-podinfo\") pod \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " Oct 30 18:03:57.055694 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.055501 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istio-envoy\") pod \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " Oct 30 18:03:57.055694 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.055591 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-workload-socket\") pod \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " Oct 30 18:03:57.055694 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.055637 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istio-token\" (UniqueName: \"kubernetes.io/projected/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istio-token\") pod \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " Oct 30 18:03:57.055938 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.055706 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istiod-ca-cert\") pod \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " Oct 30 18:03:57.055938 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.055753 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-workload-certs\") pod \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " Oct 30 18:03:57.055938 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.055890 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-kube-api-access-nldq2\" (UniqueName: \"kubernetes.io/projected/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-kube-api-access-nldq2\") pod \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " Oct 30 18:03:57.056111 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.055996 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-credential-socket\") pod \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " Oct 30 18:03:57.056111 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.056037 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istio-data\") pod \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\" (UID: \"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9\") " Oct 30 18:03:57.056209 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.056177 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-workload-certs" pod "6e3c3d72-4ad1-4368-93e4-5f429a7f80e9" (UID: "6e3c3d72-4ad1-4368-93e4-5f429a7f80e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:57.056278 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.056252 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istiod-ca-cert" pod "6e3c3d72-4ad1-4368-93e4-5f429a7f80e9" (UID: "6e3c3d72-4ad1-4368-93e4-5f429a7f80e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:03:57.056522 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.056497 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istio-data" pod "6e3c3d72-4ad1-4368-93e4-5f429a7f80e9" (UID: "6e3c3d72-4ad1-4368-93e4-5f429a7f80e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:57.056780 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.056652 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-credential-socket" pod "6e3c3d72-4ad1-4368-93e4-5f429a7f80e9" (UID: "6e3c3d72-4ad1-4368-93e4-5f429a7f80e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:57.057019 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.056964 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:03:57.057019 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.056974 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-workload-socket" pod "6e3c3d72-4ad1-4368-93e4-5f429a7f80e9" (UID: "6e3c3d72-4ad1-4368-93e4-5f429a7f80e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:57.057019 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.056994 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:03:57.057019 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.057009 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:03:57.057272 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.057025 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:03:57.067585 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.067538 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istio-envoy" pod "6e3c3d72-4ad1-4368-93e4-5f429a7f80e9" (UID: "6e3c3d72-4ad1-4368-93e4-5f429a7f80e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:57.067925 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.067840 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istio-podinfo" pod "6e3c3d72-4ad1-4368-93e4-5f429a7f80e9" (UID: "6e3c3d72-4ad1-4368-93e4-5f429a7f80e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:03:57.067925 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.067903 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istio-token" pod "6e3c3d72-4ad1-4368-93e4-5f429a7f80e9" (UID: "6e3c3d72-4ad1-4368-93e4-5f429a7f80e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:57.070417 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.070387 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-kube-api-access-nldq2" pod "6e3c3d72-4ad1-4368-93e4-5f429a7f80e9" (UID: "6e3c3d72-4ad1-4368-93e4-5f429a7f80e9"). InnerVolumeSpecName "kube-api-access-nldq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:57.159083 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.159052 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:03:57.159083 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.159080 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:03:57.159507 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.159097 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:03:57.159507 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.159109 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:03:57.159507 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.159121 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nldq2\" (UniqueName: \"kubernetes.io/projected/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9-kube-api-access-nldq2\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:03:57.449477 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.449443 4965 generic.go:356] "Generic (PLEG): container finished" podID="6e3c3d72-4ad1-4368-93e4-5f429a7f80e9" containerID="d3eb8d14fd0c7a2c882f5077197e73d9111a4619bbf7aa9152dc83749d177814" exitCode=0 Oct 30 18:03:57.449686 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.449496 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" Oct 30 18:03:57.449686 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.449526 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" event={"ID":"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9","Type":"ContainerDied","Data":"d3eb8d14fd0c7a2c882f5077197e73d9111a4619bbf7aa9152dc83749d177814"} Oct 30 18:03:57.449686 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.449587 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg" event={"ID":"6e3c3d72-4ad1-4368-93e4-5f429a7f80e9","Type":"ContainerDied","Data":"812d4ed6b1b166c075c9aea0ddc17d41f79ccdfcf42b15b1bfa0e7610f5e3bc8"} Oct 30 18:03:57.449686 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.449616 4965 scope.go:122] "RemoveContainer" containerID="d3eb8d14fd0c7a2c882f5077197e73d9111a4619bbf7aa9152dc83749d177814" Oct 30 18:03:57.462959 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.462941 4965 scope.go:122] "RemoveContainer" containerID="d3eb8d14fd0c7a2c882f5077197e73d9111a4619bbf7aa9152dc83749d177814" Oct 30 18:03:57.463311 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:03:57.463280 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3eb8d14fd0c7a2c882f5077197e73d9111a4619bbf7aa9152dc83749d177814\": container with ID starting with d3eb8d14fd0c7a2c882f5077197e73d9111a4619bbf7aa9152dc83749d177814 not found: ID does not exist" containerID="d3eb8d14fd0c7a2c882f5077197e73d9111a4619bbf7aa9152dc83749d177814" Oct 30 18:03:57.463396 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.463314 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3eb8d14fd0c7a2c882f5077197e73d9111a4619bbf7aa9152dc83749d177814"} err="failed to get container status \"d3eb8d14fd0c7a2c882f5077197e73d9111a4619bbf7aa9152dc83749d177814\": rpc error: code = NotFound desc = could not find container \"d3eb8d14fd0c7a2c882f5077197e73d9111a4619bbf7aa9152dc83749d177814\": container with ID starting with d3eb8d14fd0c7a2c882f5077197e73d9111a4619bbf7aa9152dc83749d177814 not found: ID does not exist" Oct 30 18:03:57.510622 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.510580 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg"] Oct 30 18:03:57.522219 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:57.522191 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-016f1d-openshift-default-7b66b978b9-cz7fg"] Oct 30 18:03:58.104496 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:58.104462 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e3c3d72-4ad1-4368-93e4-5f429a7f80e9" path="/var/lib/kubelet/pods/6e3c3d72-4ad1-4368-93e4-5f429a7f80e9/volumes" Oct 30 18:03:59.568096 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:59.568036 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f10dda-658f6df5fb-n9xc5"] Oct 30 18:03:59.578773 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:59.576002 4965 cpu_manager.go:414] "containerMap: removing container" podUID="844e6c1f-9e6d-4890-93e0-4216c37bc999" containerName="payload-pre-processing" Oct 30 18:03:59.578773 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:59.576033 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="844e6c1f-9e6d-4890-93e0-4216c37bc999" containerName="payload-pre-processing" Oct 30 18:03:59.578773 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:59.576068 4965 cpu_manager.go:414] "containerMap: removing container" podUID="6e3c3d72-4ad1-4368-93e4-5f429a7f80e9" containerName="istio-proxy" Oct 30 18:03:59.578773 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:59.576075 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e3c3d72-4ad1-4368-93e4-5f429a7f80e9" containerName="istio-proxy" Oct 30 18:03:59.578773 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:59.576121 4965 cpu_manager.go:414] "containerMap: removing container" podUID="e84b153b-396c-4135-bd60-a92cac969337" containerName="payload-processing" Oct 30 18:03:59.578773 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:59.576129 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e84b153b-396c-4135-bd60-a92cac969337" containerName="payload-processing" Oct 30 18:03:59.578773 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:59.576143 4965 cpu_manager.go:414] "containerMap: removing container" podUID="f4296e00-981c-4a22-8f06-d8fc997c359e" containerName="payload-processing" Oct 30 18:03:59.578773 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:59.576151 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4296e00-981c-4a22-8f06-d8fc997c359e" containerName="payload-processing" Oct 30 18:03:59.578773 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:59.576172 4965 cpu_manager.go:414] "containerMap: removing container" podUID="a23856b6-65d8-489d-9bae-299672aa44d2" containerName="payload-pre-processing" Oct 30 18:03:59.578773 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:59.576179 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a23856b6-65d8-489d-9bae-299672aa44d2" containerName="payload-pre-processing" Oct 30 18:03:59.578773 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:59.576450 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e3c3d72-4ad1-4368-93e4-5f429a7f80e9" containerName="istio-proxy" Oct 30 18:03:59.578773 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:59.576474 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="e84b153b-396c-4135-bd60-a92cac969337" containerName="payload-processing" Oct 30 18:03:59.578773 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:59.576485 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4296e00-981c-4a22-8f06-d8fc997c359e" containerName="payload-processing" Oct 30 18:03:59.578773 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:59.576497 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="a23856b6-65d8-489d-9bae-299672aa44d2" containerName="payload-pre-processing" Oct 30 18:03:59.578773 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:59.576518 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="844e6c1f-9e6d-4890-93e0-4216c37bc999" containerName="payload-pre-processing" Oct 30 18:03:59.582774 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:59.582735 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f10dda-658f6df5fb-n9xc5"] Oct 30 18:03:59.583082 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:59.582921 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-658f6df5fb-n9xc5" Oct 30 18:03:59.611666 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:59.611638 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f10dda\"" Oct 30 18:03:59.611812 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:59.611727 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f10dda-dockercfg-782rp\"" Oct 30 18:03:59.723718 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:59.723677 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b75b13e4-a0a9-45a4-84f4-87082c3d0548-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f10dda-658f6df5fb-n9xc5\" (UID: \"b75b13e4-a0a9-45a4-84f4-87082c3d0548\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-658f6df5fb-n9xc5" Oct 30 18:03:59.723948 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:59.723925 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j67fw\" (UniqueName: \"kubernetes.io/projected/b75b13e4-a0a9-45a4-84f4-87082c3d0548-kube-api-access-j67fw\") pod \"payload-processing-e2e-ipp-b-f10dda-658f6df5fb-n9xc5\" (UID: \"b75b13e4-a0a9-45a4-84f4-87082c3d0548\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-658f6df5fb-n9xc5" Oct 30 18:03:59.804798 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:59.804768 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw"] Oct 30 18:03:59.805018 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:59.804985 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw" podUID="cb5ef536-ab9a-423e-bea3-dac81549b81c" containerName="maas-api" containerID="cri-o://2fdcde90525f32ca6599745d898ab226bf785f5f8be6b47c3b253cd42f68f383" gracePeriod=30 Oct 30 18:03:59.827906 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:59.825842 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b75b13e4-a0a9-45a4-84f4-87082c3d0548-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f10dda-658f6df5fb-n9xc5\" (UID: \"b75b13e4-a0a9-45a4-84f4-87082c3d0548\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-658f6df5fb-n9xc5" Oct 30 18:03:59.827906 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:59.826072 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j67fw\" (UniqueName: \"kubernetes.io/projected/b75b13e4-a0a9-45a4-84f4-87082c3d0548-kube-api-access-j67fw\") pod \"payload-processing-e2e-ipp-b-f10dda-658f6df5fb-n9xc5\" (UID: \"b75b13e4-a0a9-45a4-84f4-87082c3d0548\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-658f6df5fb-n9xc5" Oct 30 18:03:59.827906 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:59.826637 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b75b13e4-a0a9-45a4-84f4-87082c3d0548-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f10dda-658f6df5fb-n9xc5\" (UID: \"b75b13e4-a0a9-45a4-84f4-87082c3d0548\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-658f6df5fb-n9xc5" Oct 30 18:03:59.850941 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:59.850905 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j67fw\" (UniqueName: \"kubernetes.io/projected/b75b13e4-a0a9-45a4-84f4-87082c3d0548-kube-api-access-j67fw\") pod \"payload-processing-e2e-ipp-b-f10dda-658f6df5fb-n9xc5\" (UID: \"b75b13e4-a0a9-45a4-84f4-87082c3d0548\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-658f6df5fb-n9xc5" Oct 30 18:03:59.911445 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:03:59.911414 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-658f6df5fb-n9xc5" Oct 30 18:04:00.346886 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:00.324631 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f10dda-658f6df5fb-n9xc5"] Oct 30 18:04:00.470530 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:00.465756 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw" Oct 30 18:04:00.517351 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:00.517291 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb5ef536-ab9a-423e-bea3-dac81549b81c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb5ef536-ab9a-423e-bea3-dac81549b81c-openshift-service-ca\") pod \"cb5ef536-ab9a-423e-bea3-dac81549b81c\" (UID: \"cb5ef536-ab9a-423e-bea3-dac81549b81c\") " Oct 30 18:04:00.517596 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:00.517387 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb5ef536-ab9a-423e-bea3-dac81549b81c-kube-api-access-t9szn\" (UniqueName: \"kubernetes.io/projected/cb5ef536-ab9a-423e-bea3-dac81549b81c-kube-api-access-t9szn\") pod \"cb5ef536-ab9a-423e-bea3-dac81549b81c\" (UID: \"cb5ef536-ab9a-423e-bea3-dac81549b81c\") " Oct 30 18:04:00.517596 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:00.517415 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb5ef536-ab9a-423e-bea3-dac81549b81c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb5ef536-ab9a-423e-bea3-dac81549b81c-maas-api-tls\") pod \"cb5ef536-ab9a-423e-bea3-dac81549b81c\" (UID: \"cb5ef536-ab9a-423e-bea3-dac81549b81c\") " Oct 30 18:04:00.517596 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:00.517533 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb5ef536-ab9a-423e-bea3-dac81549b81c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb5ef536-ab9a-423e-bea3-dac81549b81c-metrics-certs\") pod \"cb5ef536-ab9a-423e-bea3-dac81549b81c\" (UID: \"cb5ef536-ab9a-423e-bea3-dac81549b81c\") " Oct 30 18:04:00.517937 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:00.517669 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb5ef536-ab9a-423e-bea3-dac81549b81c-openshift-service-ca" pod "cb5ef536-ab9a-423e-bea3-dac81549b81c" (UID: "cb5ef536-ab9a-423e-bea3-dac81549b81c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:04:00.518407 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:00.518383 4965 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb5ef536-ab9a-423e-bea3-dac81549b81c-openshift-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:00.519749 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:00.519727 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb5ef536-ab9a-423e-bea3-dac81549b81c-maas-api-tls" pod "cb5ef536-ab9a-423e-bea3-dac81549b81c" (UID: "cb5ef536-ab9a-423e-bea3-dac81549b81c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:04:00.520270 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:00.520151 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb5ef536-ab9a-423e-bea3-dac81549b81c-kube-api-access-t9szn" pod "cb5ef536-ab9a-423e-bea3-dac81549b81c" (UID: "cb5ef536-ab9a-423e-bea3-dac81549b81c"). InnerVolumeSpecName "kube-api-access-t9szn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:00.520370 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:00.520353 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb5ef536-ab9a-423e-bea3-dac81549b81c-metrics-certs" pod "cb5ef536-ab9a-423e-bea3-dac81549b81c" (UID: "cb5ef536-ab9a-423e-bea3-dac81549b81c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:04:00.539913 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:00.539848 4965 generic.go:356] "Generic (PLEG): container finished" podID="cb5ef536-ab9a-423e-bea3-dac81549b81c" containerID="2fdcde90525f32ca6599745d898ab226bf785f5f8be6b47c3b253cd42f68f383" exitCode=0 Oct 30 18:04:00.540176 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:00.539921 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw" event={"ID":"cb5ef536-ab9a-423e-bea3-dac81549b81c","Type":"ContainerDied","Data":"2fdcde90525f32ca6599745d898ab226bf785f5f8be6b47c3b253cd42f68f383"} Oct 30 18:04:00.540303 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:00.540199 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw" event={"ID":"cb5ef536-ab9a-423e-bea3-dac81549b81c","Type":"ContainerDied","Data":"438d4d0d281bedd35410d69bbb9c5906478c36f004d7cee7a9d9a26ca2139d97"} Oct 30 18:04:00.540303 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:00.539939 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw" Oct 30 18:04:00.540482 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:00.540227 4965 scope.go:122] "RemoveContainer" containerID="2fdcde90525f32ca6599745d898ab226bf785f5f8be6b47c3b253cd42f68f383" Oct 30 18:04:00.543230 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:00.543202 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-658f6df5fb-n9xc5" event={"ID":"b75b13e4-a0a9-45a4-84f4-87082c3d0548","Type":"ContainerStarted","Data":"39689d8deb14fc77fe4c116226198de58fc9c8485778b140b20b41dfc061d81d"} Oct 30 18:04:00.569975 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:00.569593 4965 scope.go:122] "RemoveContainer" containerID="2fdcde90525f32ca6599745d898ab226bf785f5f8be6b47c3b253cd42f68f383" Oct 30 18:04:00.572724 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:04:00.572694 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fdcde90525f32ca6599745d898ab226bf785f5f8be6b47c3b253cd42f68f383\": container with ID starting with 2fdcde90525f32ca6599745d898ab226bf785f5f8be6b47c3b253cd42f68f383 not found: ID does not exist" containerID="2fdcde90525f32ca6599745d898ab226bf785f5f8be6b47c3b253cd42f68f383" Oct 30 18:04:00.573002 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:00.572957 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fdcde90525f32ca6599745d898ab226bf785f5f8be6b47c3b253cd42f68f383"} err="failed to get container status \"2fdcde90525f32ca6599745d898ab226bf785f5f8be6b47c3b253cd42f68f383\": rpc error: code = NotFound desc = could not find container \"2fdcde90525f32ca6599745d898ab226bf785f5f8be6b47c3b253cd42f68f383\": container with ID starting with 2fdcde90525f32ca6599745d898ab226bf785f5f8be6b47c3b253cd42f68f383 not found: ID does not exist" Oct 30 18:04:00.595374 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:00.595338 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw"] Oct 30 18:04:00.600367 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:00.600337 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-90b48a-5f9bc7948b-w8rmw"] Oct 30 18:04:00.620092 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:00.620062 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9szn\" (UniqueName: \"kubernetes.io/projected/cb5ef536-ab9a-423e-bea3-dac81549b81c-kube-api-access-t9szn\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:00.620092 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:00.620093 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb5ef536-ab9a-423e-bea3-dac81549b81c-maas-api-tls\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:00.620295 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:00.620109 4965 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb5ef536-ab9a-423e-bea3-dac81549b81c-metrics-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:01.553439 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:01.553393 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-658f6df5fb-n9xc5" event={"ID":"b75b13e4-a0a9-45a4-84f4-87082c3d0548","Type":"ContainerStarted","Data":"7576743247ef5fc41fdcdf395b26877644743958a071da61f67bddef3d2c7007"} Oct 30 18:04:01.553647 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:01.553498 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-658f6df5fb-n9xc5" Oct 30 18:04:01.597631 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:01.597582 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-658f6df5fb-n9xc5" podStartSLOduration=2.5975669630000002 podStartE2EDuration="2.597566963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:01.594042541 +0000 UTC m=+3099.646142473" watchObservedRunningTime="2026-10-30 18:04:01.597566963 +0000 UTC m=+3099.649666894" Oct 30 18:04:02.103229 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:02.103194 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb5ef536-ab9a-423e-bea3-dac81549b81c" path="/var/lib/kubelet/pods/cb5ef536-ab9a-423e-bea3-dac81549b81c/volumes" Oct 30 18:04:05.997565 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:04:05.997484 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:04:06.533471 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:06.533440 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ce0480-5cd9bdc47d-974w6"] Oct 30 18:04:08.087447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:08.087414 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs"] Oct 30 18:04:08.087880 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:08.087619 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" podUID="2edd6ff6-320a-42d4-83ef-5173fcbf7b92" containerName="istio-proxy" containerID="cri-o://1baca4c4e7f67345f25459682c044f9aec02944e4e8c4be5efa1a5b8b3510621" gracePeriod=30 Oct 30 18:04:11.534601 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:11.534557 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ce0480-5cd9bdc47d-974w6" podUID="f2721707-097c-4adf-a415-15be655b9729" containerName="payload-processing" containerID="cri-o://a3fb17e3d2c78d0626a185335a275d8083e458574994cae161f955e27803170a" gracePeriod=25 Oct 30 18:04:12.564895 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:12.564831 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-658f6df5fb-n9xc5" Oct 30 18:04:13.440940 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.440906 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 18:04:13.556132 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.556098 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istio-data\") pod \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " Oct 30 18:04:13.556385 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.556160 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istio-podinfo\") pod \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " Oct 30 18:04:13.556385 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.556372 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-credential-socket\") pod \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " Oct 30 18:04:13.556504 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.556412 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-workload-socket\") pod \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " Oct 30 18:04:13.556504 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.556419 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istio-data" pod "2edd6ff6-320a-42d4-83ef-5173fcbf7b92" (UID: "2edd6ff6-320a-42d4-83ef-5173fcbf7b92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:13.556611 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.556548 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-kube-api-access-2k2dh\" (UniqueName: \"kubernetes.io/projected/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-kube-api-access-2k2dh\") pod \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " Oct 30 18:04:13.556611 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.556601 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istiod-ca-cert\") pod \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " Oct 30 18:04:13.556728 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.556641 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istio-envoy\") pod \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " Oct 30 18:04:13.556728 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.556695 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-credential-socket" pod "2edd6ff6-320a-42d4-83ef-5173fcbf7b92" (UID: "2edd6ff6-320a-42d4-83ef-5173fcbf7b92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:13.556728 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.556708 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-workload-socket" pod "2edd6ff6-320a-42d4-83ef-5173fcbf7b92" (UID: "2edd6ff6-320a-42d4-83ef-5173fcbf7b92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:13.556928 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.556756 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istio-token\" (UniqueName: \"kubernetes.io/projected/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istio-token\") pod \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " Oct 30 18:04:13.556928 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.556780 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-workload-certs\") pod \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\" (UID: \"2edd6ff6-320a-42d4-83ef-5173fcbf7b92\") " Oct 30 18:04:13.557164 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.557129 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-workload-certs" pod "2edd6ff6-320a-42d4-83ef-5173fcbf7b92" (UID: "2edd6ff6-320a-42d4-83ef-5173fcbf7b92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:13.557256 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.557219 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istiod-ca-cert" pod "2edd6ff6-320a-42d4-83ef-5173fcbf7b92" (UID: "2edd6ff6-320a-42d4-83ef-5173fcbf7b92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:04:13.557681 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.557658 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:13.557853 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.557685 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:13.557853 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.557698 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:13.557853 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.557711 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:13.557853 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.557725 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:13.558786 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.558764 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istio-podinfo" pod "2edd6ff6-320a-42d4-83ef-5173fcbf7b92" (UID: "2edd6ff6-320a-42d4-83ef-5173fcbf7b92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:04:13.558930 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.558788 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-kube-api-access-2k2dh" pod "2edd6ff6-320a-42d4-83ef-5173fcbf7b92" (UID: "2edd6ff6-320a-42d4-83ef-5173fcbf7b92"). InnerVolumeSpecName "kube-api-access-2k2dh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:13.559014 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.558977 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istio-token" pod "2edd6ff6-320a-42d4-83ef-5173fcbf7b92" (UID: "2edd6ff6-320a-42d4-83ef-5173fcbf7b92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:13.559059 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.559028 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istio-envoy" pod "2edd6ff6-320a-42d4-83ef-5173fcbf7b92" (UID: "2edd6ff6-320a-42d4-83ef-5173fcbf7b92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:13.659113 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.659031 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k2dh\" (UniqueName: \"kubernetes.io/projected/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-kube-api-access-2k2dh\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:13.659113 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.659063 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:13.659113 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.659073 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:13.659113 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.659083 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2edd6ff6-320a-42d4-83ef-5173fcbf7b92-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:13.673855 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.673826 4965 generic.go:356] "Generic (PLEG): container finished" podID="2edd6ff6-320a-42d4-83ef-5173fcbf7b92" containerID="1baca4c4e7f67345f25459682c044f9aec02944e4e8c4be5efa1a5b8b3510621" exitCode=0 Oct 30 18:04:13.674016 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.673895 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" event={"ID":"2edd6ff6-320a-42d4-83ef-5173fcbf7b92","Type":"ContainerDied","Data":"1baca4c4e7f67345f25459682c044f9aec02944e4e8c4be5efa1a5b8b3510621"} Oct 30 18:04:13.674016 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.673912 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" Oct 30 18:04:13.674016 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.673962 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs" event={"ID":"2edd6ff6-320a-42d4-83ef-5173fcbf7b92","Type":"ContainerDied","Data":"8fdb075ba5a10f8cf7afc4007442b1f0285f8f047cd9253ebd364a284567c977"} Oct 30 18:04:13.674016 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.673992 4965 scope.go:122] "RemoveContainer" containerID="1baca4c4e7f67345f25459682c044f9aec02944e4e8c4be5efa1a5b8b3510621" Oct 30 18:04:13.689039 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.688942 4965 scope.go:122] "RemoveContainer" containerID="0a653659716682526a9f79db0df47f6c6b0b5aa5119f56d2ad6ada9fd7fda922" Oct 30 18:04:13.698986 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.698967 4965 scope.go:122] "RemoveContainer" containerID="1baca4c4e7f67345f25459682c044f9aec02944e4e8c4be5efa1a5b8b3510621" Oct 30 18:04:13.699275 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:04:13.699247 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1baca4c4e7f67345f25459682c044f9aec02944e4e8c4be5efa1a5b8b3510621\": container with ID starting with 1baca4c4e7f67345f25459682c044f9aec02944e4e8c4be5efa1a5b8b3510621 not found: ID does not exist" containerID="1baca4c4e7f67345f25459682c044f9aec02944e4e8c4be5efa1a5b8b3510621" Oct 30 18:04:13.699335 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.699283 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1baca4c4e7f67345f25459682c044f9aec02944e4e8c4be5efa1a5b8b3510621"} err="failed to get container status \"1baca4c4e7f67345f25459682c044f9aec02944e4e8c4be5efa1a5b8b3510621\": rpc error: code = NotFound desc = could not find container \"1baca4c4e7f67345f25459682c044f9aec02944e4e8c4be5efa1a5b8b3510621\": container with ID starting with 1baca4c4e7f67345f25459682c044f9aec02944e4e8c4be5efa1a5b8b3510621 not found: ID does not exist" Oct 30 18:04:13.699335 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.699300 4965 scope.go:122] "RemoveContainer" containerID="0a653659716682526a9f79db0df47f6c6b0b5aa5119f56d2ad6ada9fd7fda922" Oct 30 18:04:13.699544 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:04:13.699525 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a653659716682526a9f79db0df47f6c6b0b5aa5119f56d2ad6ada9fd7fda922\": container with ID starting with 0a653659716682526a9f79db0df47f6c6b0b5aa5119f56d2ad6ada9fd7fda922 not found: ID does not exist" containerID="0a653659716682526a9f79db0df47f6c6b0b5aa5119f56d2ad6ada9fd7fda922" Oct 30 18:04:13.699608 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.699553 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a653659716682526a9f79db0df47f6c6b0b5aa5119f56d2ad6ada9fd7fda922"} err="failed to get container status \"0a653659716682526a9f79db0df47f6c6b0b5aa5119f56d2ad6ada9fd7fda922\": rpc error: code = NotFound desc = could not find container \"0a653659716682526a9f79db0df47f6c6b0b5aa5119f56d2ad6ada9fd7fda922\": container with ID starting with 0a653659716682526a9f79db0df47f6c6b0b5aa5119f56d2ad6ada9fd7fda922 not found: ID does not exist" Oct 30 18:04:13.728976 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.728948 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs"] Oct 30 18:04:13.740568 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:13.740542 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-90b48a-openshift-default-8576f6d8f-ckhqs"] Oct 30 18:04:14.101245 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:14.101214 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2edd6ff6-320a-42d4-83ef-5173fcbf7b92" path="/var/lib/kubelet/pods/2edd6ff6-320a-42d4-83ef-5173fcbf7b92/volumes" Oct 30 18:04:16.397799 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:16.397757 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ce0480-5cd9bdc47d-974w6" podUID="f2721707-097c-4adf-a415-15be655b9729" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:04:17.810830 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:17.810798 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-5tdrf_d383c0b3-9245-4397-b031-38295a768fba/payload-processing/0.log" Oct 30 18:04:18.011768 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:18.011738 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-5tdrf_d383c0b3-9245-4397-b031-38295a768fba/payload-processing/0.log" Oct 30 18:04:18.206744 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:18.206711 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f10dda-658f6df5fb-n9xc5_b75b13e4-a0a9-45a4-84f4-87082c3d0548/payload-processing/0.log" Oct 30 18:04:19.178130 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:19.178094 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-738845-867b76d77d-qzjng"] Oct 30 18:04:19.209766 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:19.209737 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-738845-6c477d4fb7-r4tnv"] Oct 30 18:04:24.178778 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:24.178732 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-738845-867b76d77d-qzjng" podUID="8abc8140-8a36-4cc7-80da-43738e049fe0" containerName="payload-processing" containerID="cri-o://c5ab5ce4cb9fe7248cb0a7c171ed70c94540182e0a81bd61540f805ddcff88c2" gracePeriod=25 Oct 30 18:04:24.210543 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:24.210498 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-738845-6c477d4fb7-r4tnv" podUID="dcb3f0ba-021a-4788-ad64-fcc9054fe5a1" containerName="payload-pre-processing" containerID="cri-o://2055b4612290f8e5685b3f98cdaf2f5bdb8b5801daedccae1ee5434ef1633b03" gracePeriod=25 Oct 30 18:04:25.639234 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:25.639197 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-738845-6c477d4fb7-r4tnv" podUID="dcb3f0ba-021a-4788-ad64-fcc9054fe5a1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:04:25.639656 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:25.639197 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-738845-867b76d77d-qzjng" podUID="8abc8140-8a36-4cc7-80da-43738e049fe0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:04:26.403414 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:26.403366 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ce0480-5cd9bdc47d-974w6" podUID="f2721707-097c-4adf-a415-15be655b9729" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:04:28.487139 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:28.487101 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-455e4a-6d96cfd86b-kmrsv"] Oct 30 18:04:28.487548 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:28.487332 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-6d96cfd86b-kmrsv" podUID="4bd72d29-e1f9-49cd-8388-156d3796563a" containerName="payload-processing" containerID="cri-o://bcc554a350e0719015bd25b874f26c48236a8b484aef03259a575bc0dee24e1f" gracePeriod=30 Oct 30 18:04:28.838718 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:28.838686 4965 generic.go:356] "Generic (PLEG): container finished" podID="4bd72d29-e1f9-49cd-8388-156d3796563a" containerID="bcc554a350e0719015bd25b874f26c48236a8b484aef03259a575bc0dee24e1f" exitCode=0 Oct 30 18:04:28.838991 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:28.838971 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-6d96cfd86b-kmrsv" event={"ID":"4bd72d29-e1f9-49cd-8388-156d3796563a","Type":"ContainerDied","Data":"bcc554a350e0719015bd25b874f26c48236a8b484aef03259a575bc0dee24e1f"} Oct 30 18:04:28.839086 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:28.839027 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-6d96cfd86b-kmrsv" event={"ID":"4bd72d29-e1f9-49cd-8388-156d3796563a","Type":"ContainerDied","Data":"51844622ee9814b9f6ab39e5c549f99c4093565a6a642b8126c64910a67d4125"} Oct 30 18:04:28.839086 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:28.839054 4965 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51844622ee9814b9f6ab39e5c549f99c4093565a6a642b8126c64910a67d4125" Oct 30 18:04:28.940883 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:28.931268 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-6d96cfd86b-kmrsv" Oct 30 18:04:29.076946 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:29.071460 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bd72d29-e1f9-49cd-8388-156d3796563a-kube-api-access-2mtjf\" (UniqueName: \"kubernetes.io/projected/4bd72d29-e1f9-49cd-8388-156d3796563a-kube-api-access-2mtjf\") pod \"4bd72d29-e1f9-49cd-8388-156d3796563a\" (UID: \"4bd72d29-e1f9-49cd-8388-156d3796563a\") " Oct 30 18:04:29.076946 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:29.071750 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bd72d29-e1f9-49cd-8388-156d3796563a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd72d29-e1f9-49cd-8388-156d3796563a-plugins-config-volume\") pod \"4bd72d29-e1f9-49cd-8388-156d3796563a\" (UID: \"4bd72d29-e1f9-49cd-8388-156d3796563a\") " Oct 30 18:04:29.098948 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:29.088304 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bd72d29-e1f9-49cd-8388-156d3796563a-plugins-config-volume" pod "4bd72d29-e1f9-49cd-8388-156d3796563a" (UID: "4bd72d29-e1f9-49cd-8388-156d3796563a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:04:29.175620 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:29.175577 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd72d29-e1f9-49cd-8388-156d3796563a-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:29.176965 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:29.176917 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bd72d29-e1f9-49cd-8388-156d3796563a-kube-api-access-2mtjf" pod "4bd72d29-e1f9-49cd-8388-156d3796563a" (UID: "4bd72d29-e1f9-49cd-8388-156d3796563a"). InnerVolumeSpecName "kube-api-access-2mtjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:29.285531 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:29.285474 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mtjf\" (UniqueName: \"kubernetes.io/projected/4bd72d29-e1f9-49cd-8388-156d3796563a-kube-api-access-2mtjf\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:29.849714 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:29.849682 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-455e4a-6d96cfd86b-kmrsv" Oct 30 18:04:29.923479 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:29.923441 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-455e4a-6d96cfd86b-kmrsv"] Oct 30 18:04:29.927926 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:29.927884 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-455e4a-6d96cfd86b-kmrsv"] Oct 30 18:04:30.101750 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:30.101663 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bd72d29-e1f9-49cd-8388-156d3796563a" path="/var/lib/kubelet/pods/4bd72d29-e1f9-49cd-8388-156d3796563a/volumes" Oct 30 18:04:31.848074 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:31.848045 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ce0480-5cd9bdc47d-974w6" Oct 30 18:04:31.886847 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:31.886806 4965 generic.go:356] "Generic (PLEG): container finished" podID="f2721707-097c-4adf-a415-15be655b9729" containerID="a3fb17e3d2c78d0626a185335a275d8083e458574994cae161f955e27803170a" exitCode=0 Oct 30 18:04:31.887067 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:31.886890 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ce0480-5cd9bdc47d-974w6" event={"ID":"f2721707-097c-4adf-a415-15be655b9729","Type":"ContainerDied","Data":"a3fb17e3d2c78d0626a185335a275d8083e458574994cae161f955e27803170a"} Oct 30 18:04:31.887067 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:31.886853 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ce0480-5cd9bdc47d-974w6" Oct 30 18:04:31.887067 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:31.886947 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ce0480-5cd9bdc47d-974w6" event={"ID":"f2721707-097c-4adf-a415-15be655b9729","Type":"ContainerDied","Data":"08172d9e3001fdbcb59b03a3f02a408c6f41d774e143acf48500d20eaad8d1c6"} Oct 30 18:04:31.887067 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:31.886972 4965 scope.go:122] "RemoveContainer" containerID="a3fb17e3d2c78d0626a185335a275d8083e458574994cae161f955e27803170a" Oct 30 18:04:31.901775 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:31.901748 4965 scope.go:122] "RemoveContainer" containerID="a3fb17e3d2c78d0626a185335a275d8083e458574994cae161f955e27803170a" Oct 30 18:04:31.902203 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:04:31.902146 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3fb17e3d2c78d0626a185335a275d8083e458574994cae161f955e27803170a\": container with ID starting with a3fb17e3d2c78d0626a185335a275d8083e458574994cae161f955e27803170a not found: ID does not exist" containerID="a3fb17e3d2c78d0626a185335a275d8083e458574994cae161f955e27803170a" Oct 30 18:04:31.902203 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:31.902178 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3fb17e3d2c78d0626a185335a275d8083e458574994cae161f955e27803170a"} err="failed to get container status \"a3fb17e3d2c78d0626a185335a275d8083e458574994cae161f955e27803170a\": rpc error: code = NotFound desc = could not find container \"a3fb17e3d2c78d0626a185335a275d8083e458574994cae161f955e27803170a\": container with ID starting with a3fb17e3d2c78d0626a185335a275d8083e458574994cae161f955e27803170a not found: ID does not exist" Oct 30 18:04:31.951492 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:31.951456 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2721707-097c-4adf-a415-15be655b9729-kube-api-access-zt868\" (UniqueName: \"kubernetes.io/projected/f2721707-097c-4adf-a415-15be655b9729-kube-api-access-zt868\") pod \"f2721707-097c-4adf-a415-15be655b9729\" (UID: \"f2721707-097c-4adf-a415-15be655b9729\") " Oct 30 18:04:31.951768 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:31.951749 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2721707-097c-4adf-a415-15be655b9729-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2721707-097c-4adf-a415-15be655b9729-plugins-config-volume\") pod \"f2721707-097c-4adf-a415-15be655b9729\" (UID: \"f2721707-097c-4adf-a415-15be655b9729\") " Oct 30 18:04:31.952260 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:31.952228 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2721707-097c-4adf-a415-15be655b9729-plugins-config-volume" pod "f2721707-097c-4adf-a415-15be655b9729" (UID: "f2721707-097c-4adf-a415-15be655b9729"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:04:31.952505 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:31.952484 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2721707-097c-4adf-a415-15be655b9729-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:31.954023 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:31.953998 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2721707-097c-4adf-a415-15be655b9729-kube-api-access-zt868" pod "f2721707-097c-4adf-a415-15be655b9729" (UID: "f2721707-097c-4adf-a415-15be655b9729"). InnerVolumeSpecName "kube-api-access-zt868". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:32.053815 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:32.053774 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt868\" (UniqueName: \"kubernetes.io/projected/f2721707-097c-4adf-a415-15be655b9729-kube-api-access-zt868\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:32.229255 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:32.229216 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ce0480-5cd9bdc47d-974w6"] Oct 30 18:04:32.234670 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:32.234640 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ce0480-5cd9bdc47d-974w6"] Oct 30 18:04:34.100724 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:34.100689 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2721707-097c-4adf-a415-15be655b9729" path="/var/lib/kubelet/pods/f2721707-097c-4adf-a415-15be655b9729/volumes" Oct 30 18:04:35.688722 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:35.685643 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-738845-6c477d4fb7-r4tnv" podUID="dcb3f0ba-021a-4788-ad64-fcc9054fe5a1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:04:35.759266 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:35.752036 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-738845-867b76d77d-qzjng" podUID="8abc8140-8a36-4cc7-80da-43738e049fe0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:04:35.777033 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:35.776992 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn"] Oct 30 18:04:35.777392 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:35.777359 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" podUID="b48e4e05-e95d-424c-887e-668bdc42e398" containerName="istio-proxy" containerID="cri-o://b9c4f58f889a7e1a4a99f5438229707c75b52e7b3780b12d599a8f6b27e97649" gracePeriod=30 Oct 30 18:04:36.077351 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:04:36.077312 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:04:36.965347 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:36.965311 4965 generic.go:356] "Generic (PLEG): container finished" podID="4a2a43ad-860d-465a-a29a-130c83307e87" containerID="bca2e1151789163153b4d37b2637190d8e61196448ac81819f62c347e45ca03e" exitCode=0 Oct 30 18:04:36.965744 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:36.965384 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" event={"ID":"4a2a43ad-860d-465a-a29a-130c83307e87","Type":"ContainerDied","Data":"bca2e1151789163153b4d37b2637190d8e61196448ac81819f62c347e45ca03e"} Oct 30 18:04:36.965846 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:36.965832 4965 scope.go:122] "RemoveContainer" containerID="bca2e1151789163153b4d37b2637190d8e61196448ac81819f62c347e45ca03e" Oct 30 18:04:36.967666 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:36.967636 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 18:04:36.967757 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:36.967706 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 18:04:36.967757 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:36.967748 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 18:04:37.393794 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:37.393759 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:04:37.394052 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:37.393817 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" podUID="b48e4e05-e95d-424c-887e-668bdc42e398" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:04:37.975165 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:37.975132 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" event={"ID":"4a2a43ad-860d-465a-a29a-130c83307e87","Type":"ContainerStarted","Data":"a086d9be149ff5d96c509f945f3ff144fe40962fe614676050145b9edecaff66"} Oct 30 18:04:38.808930 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:38.808893 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:04:38.809113 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:38.809091 4965 patch_prober.go:29] interesting pod/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 18:04:38.809173 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:38.809140 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" podUID="4a2a43ad-860d-465a-a29a-130c83307e87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 18:04:39.809212 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:39.809170 4965 patch_prober.go:29] interesting pod/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 18:04:39.809625 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:39.809249 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" podUID="4a2a43ad-860d-465a-a29a-130c83307e87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 18:04:40.809158 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:40.809124 4965 patch_prober.go:29] interesting pod/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 18:04:40.809277 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:40.809192 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" podUID="4a2a43ad-860d-465a-a29a-130c83307e87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 18:04:41.020213 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.020176 4965 generic.go:356] "Generic (PLEG): container finished" podID="b48e4e05-e95d-424c-887e-668bdc42e398" containerID="b9c4f58f889a7e1a4a99f5438229707c75b52e7b3780b12d599a8f6b27e97649" exitCode=0 Oct 30 18:04:41.020785 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.020363 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" event={"ID":"b48e4e05-e95d-424c-887e-668bdc42e398","Type":"ContainerDied","Data":"b9c4f58f889a7e1a4a99f5438229707c75b52e7b3780b12d599a8f6b27e97649"} Oct 30 18:04:41.020785 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.020660 4965 scope.go:122] "RemoveContainer" containerID="c496ac3cc83d4845fddb5f3a7d2d11aba86a1c1693c78d96f222c97494166752" Oct 30 18:04:41.130692 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.130669 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 18:04:41.251589 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.251557 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-credential-socket\") pod \"b48e4e05-e95d-424c-887e-668bdc42e398\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " Oct 30 18:04:41.251794 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.251719 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-istio-data\") pod \"b48e4e05-e95d-424c-887e-668bdc42e398\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " Oct 30 18:04:41.251794 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.251762 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-istio-envoy\") pod \"b48e4e05-e95d-424c-887e-668bdc42e398\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " Oct 30 18:04:41.251794 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.251790 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b48e4e05-e95d-424c-887e-668bdc42e398-istio-token\" (UniqueName: \"kubernetes.io/projected/b48e4e05-e95d-424c-887e-668bdc42e398-istio-token\") pod \"b48e4e05-e95d-424c-887e-668bdc42e398\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " Oct 30 18:04:41.252012 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.251827 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-workload-socket\") pod \"b48e4e05-e95d-424c-887e-668bdc42e398\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " Oct 30 18:04:41.252012 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.251847 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-workload-certs\") pod \"b48e4e05-e95d-424c-887e-668bdc42e398\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " Oct 30 18:04:41.252222 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.252199 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b48e4e05-e95d-424c-887e-668bdc42e398-kube-api-access-ddgrr\" (UniqueName: \"kubernetes.io/projected/b48e4e05-e95d-424c-887e-668bdc42e398-kube-api-access-ddgrr\") pod \"b48e4e05-e95d-424c-887e-668bdc42e398\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " Oct 30 18:04:41.252297 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.252244 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-credential-socket" pod "b48e4e05-e95d-424c-887e-668bdc42e398" (UID: "b48e4e05-e95d-424c-887e-668bdc42e398"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:41.252363 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.252329 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b48e4e05-e95d-424c-887e-668bdc42e398-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48e4e05-e95d-424c-887e-668bdc42e398-istio-podinfo\") pod \"b48e4e05-e95d-424c-887e-668bdc42e398\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " Oct 30 18:04:41.252428 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.252390 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b48e4e05-e95d-424c-887e-668bdc42e398-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48e4e05-e95d-424c-887e-668bdc42e398-istiod-ca-cert\") pod \"b48e4e05-e95d-424c-887e-668bdc42e398\" (UID: \"b48e4e05-e95d-424c-887e-668bdc42e398\") " Oct 30 18:04:41.252487 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.252434 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-workload-certs" pod "b48e4e05-e95d-424c-887e-668bdc42e398" (UID: "b48e4e05-e95d-424c-887e-668bdc42e398"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:41.252487 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.252453 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-workload-socket" pod "b48e4e05-e95d-424c-887e-668bdc42e398" (UID: "b48e4e05-e95d-424c-887e-668bdc42e398"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:41.252604 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.252578 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-istio-data" pod "b48e4e05-e95d-424c-887e-668bdc42e398" (UID: "b48e4e05-e95d-424c-887e-668bdc42e398"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:41.253164 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.252944 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b48e4e05-e95d-424c-887e-668bdc42e398-istiod-ca-cert" pod "b48e4e05-e95d-424c-887e-668bdc42e398" (UID: "b48e4e05-e95d-424c-887e-668bdc42e398"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:04:41.253164 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.253069 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:41.253164 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.253091 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:41.253164 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.253106 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:41.253164 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.253120 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:41.253164 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.253133 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48e4e05-e95d-424c-887e-668bdc42e398-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:41.254808 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.254781 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b48e4e05-e95d-424c-887e-668bdc42e398-istio-token" pod "b48e4e05-e95d-424c-887e-668bdc42e398" (UID: "b48e4e05-e95d-424c-887e-668bdc42e398"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:41.255960 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.255918 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b48e4e05-e95d-424c-887e-668bdc42e398-kube-api-access-ddgrr" pod "b48e4e05-e95d-424c-887e-668bdc42e398" (UID: "b48e4e05-e95d-424c-887e-668bdc42e398"). InnerVolumeSpecName "kube-api-access-ddgrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:41.256484 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.256457 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-istio-envoy" pod "b48e4e05-e95d-424c-887e-668bdc42e398" (UID: "b48e4e05-e95d-424c-887e-668bdc42e398"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:41.290039 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.288151 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b48e4e05-e95d-424c-887e-668bdc42e398-istio-podinfo" pod "b48e4e05-e95d-424c-887e-668bdc42e398" (UID: "b48e4e05-e95d-424c-887e-668bdc42e398"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:04:41.354287 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.354253 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48e4e05-e95d-424c-887e-668bdc42e398-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:41.354287 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.354282 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b48e4e05-e95d-424c-887e-668bdc42e398-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:41.354287 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.354292 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddgrr\" (UniqueName: \"kubernetes.io/projected/b48e4e05-e95d-424c-887e-668bdc42e398-kube-api-access-ddgrr\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:41.354540 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.354301 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48e4e05-e95d-424c-887e-668bdc42e398-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:41.809349 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.809260 4965 patch_prober.go:29] interesting pod/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 18:04:41.809349 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:41.809329 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" podUID="4a2a43ad-860d-465a-a29a-130c83307e87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 18:04:42.029596 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:42.029569 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" Oct 30 18:04:42.029783 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:42.029608 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn" event={"ID":"b48e4e05-e95d-424c-887e-668bdc42e398","Type":"ContainerDied","Data":"b6c2783031b1519795c2236a8ec21904c738fa1faf203deab5fdb65c909790a6"} Oct 30 18:04:42.029783 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:42.029668 4965 scope.go:122] "RemoveContainer" containerID="b9c4f58f889a7e1a4a99f5438229707c75b52e7b3780b12d599a8f6b27e97649" Oct 30 18:04:42.103725 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:42.103679 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn"] Oct 30 18:04:42.115332 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:42.115299 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ce0480-openshift-default-86bfdd4d49-6g4nn"] Oct 30 18:04:42.809099 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:42.809067 4965 patch_prober.go:29] interesting pod/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 18:04:42.809297 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:42.809126 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" podUID="4a2a43ad-860d-465a-a29a-130c83307e87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 18:04:42.993098 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:42.993065 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:04:43.842635 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:43.842604 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:04:43.843201 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:43.843179 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:04:43.844570 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:43.844542 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:04:44.100698 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:44.100611 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b48e4e05-e95d-424c-887e-668bdc42e398" path="/var/lib/kubelet/pods/b48e4e05-e95d-424c-887e-668bdc42e398/volumes" Oct 30 18:04:44.888223 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:44.888187 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-738845-867b76d77d-qzjng" Oct 30 18:04:44.943109 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:44.943081 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-738845-6c477d4fb7-r4tnv" Oct 30 18:04:44.950456 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:44.950418 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8abc8140-8a36-4cc7-80da-43738e049fe0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8abc8140-8a36-4cc7-80da-43738e049fe0-plugins-config-volume\") pod \"8abc8140-8a36-4cc7-80da-43738e049fe0\" (UID: \"8abc8140-8a36-4cc7-80da-43738e049fe0\") " Oct 30 18:04:44.950616 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:44.950543 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8abc8140-8a36-4cc7-80da-43738e049fe0-kube-api-access-bdxfr\" (UniqueName: \"kubernetes.io/projected/8abc8140-8a36-4cc7-80da-43738e049fe0-kube-api-access-bdxfr\") pod \"8abc8140-8a36-4cc7-80da-43738e049fe0\" (UID: \"8abc8140-8a36-4cc7-80da-43738e049fe0\") " Oct 30 18:04:44.950780 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:44.950757 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8abc8140-8a36-4cc7-80da-43738e049fe0-plugins-config-volume" pod "8abc8140-8a36-4cc7-80da-43738e049fe0" (UID: "8abc8140-8a36-4cc7-80da-43738e049fe0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:04:44.951072 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:44.951056 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8abc8140-8a36-4cc7-80da-43738e049fe0-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:44.953322 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:44.953291 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8abc8140-8a36-4cc7-80da-43738e049fe0-kube-api-access-bdxfr" pod "8abc8140-8a36-4cc7-80da-43738e049fe0" (UID: "8abc8140-8a36-4cc7-80da-43738e049fe0"). InnerVolumeSpecName "kube-api-access-bdxfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:45.076607 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:45.065154 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcb3f0ba-021a-4788-ad64-fcc9054fe5a1-kube-api-access-w8smb\" (UniqueName: \"kubernetes.io/projected/dcb3f0ba-021a-4788-ad64-fcc9054fe5a1-kube-api-access-w8smb\") pod \"dcb3f0ba-021a-4788-ad64-fcc9054fe5a1\" (UID: \"dcb3f0ba-021a-4788-ad64-fcc9054fe5a1\") " Oct 30 18:04:45.076607 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:45.065481 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcb3f0ba-021a-4788-ad64-fcc9054fe5a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb3f0ba-021a-4788-ad64-fcc9054fe5a1-plugins-config-volume\") pod \"dcb3f0ba-021a-4788-ad64-fcc9054fe5a1\" (UID: \"dcb3f0ba-021a-4788-ad64-fcc9054fe5a1\") " Oct 30 18:04:45.076607 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:45.066218 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdxfr\" (UniqueName: \"kubernetes.io/projected/8abc8140-8a36-4cc7-80da-43738e049fe0-kube-api-access-bdxfr\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:45.076607 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:45.066757 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcb3f0ba-021a-4788-ad64-fcc9054fe5a1-plugins-config-volume" pod "dcb3f0ba-021a-4788-ad64-fcc9054fe5a1" (UID: "dcb3f0ba-021a-4788-ad64-fcc9054fe5a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:04:45.153949 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:45.144145 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcb3f0ba-021a-4788-ad64-fcc9054fe5a1-kube-api-access-w8smb" pod "dcb3f0ba-021a-4788-ad64-fcc9054fe5a1" (UID: "dcb3f0ba-021a-4788-ad64-fcc9054fe5a1"). InnerVolumeSpecName "kube-api-access-w8smb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:45.153949 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:45.153828 4965 generic.go:356] "Generic (PLEG): container finished" podID="8abc8140-8a36-4cc7-80da-43738e049fe0" containerID="c5ab5ce4cb9fe7248cb0a7c171ed70c94540182e0a81bd61540f805ddcff88c2" exitCode=0 Oct 30 18:04:45.154503 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:45.154137 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-738845-867b76d77d-qzjng" event={"ID":"8abc8140-8a36-4cc7-80da-43738e049fe0","Type":"ContainerDied","Data":"c5ab5ce4cb9fe7248cb0a7c171ed70c94540182e0a81bd61540f805ddcff88c2"} Oct 30 18:04:45.154503 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:45.154214 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-738845-867b76d77d-qzjng" event={"ID":"8abc8140-8a36-4cc7-80da-43738e049fe0","Type":"ContainerDied","Data":"ce370e1c1426bbeaf85c3273630fe4c0307a90fa6f2f8ac906e3c5501248524a"} Oct 30 18:04:45.154503 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:45.154273 4965 scope.go:122] "RemoveContainer" containerID="c5ab5ce4cb9fe7248cb0a7c171ed70c94540182e0a81bd61540f805ddcff88c2" Oct 30 18:04:45.154503 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:45.154460 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-738845-867b76d77d-qzjng" Oct 30 18:04:45.187191 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:45.158465 4965 generic.go:356] "Generic (PLEG): container finished" podID="dcb3f0ba-021a-4788-ad64-fcc9054fe5a1" containerID="2055b4612290f8e5685b3f98cdaf2f5bdb8b5801daedccae1ee5434ef1633b03" exitCode=0 Oct 30 18:04:45.187191 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:45.159557 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-738845-6c477d4fb7-r4tnv" Oct 30 18:04:45.187191 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:45.167191 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-738845-6c477d4fb7-r4tnv" event={"ID":"dcb3f0ba-021a-4788-ad64-fcc9054fe5a1","Type":"ContainerDied","Data":"2055b4612290f8e5685b3f98cdaf2f5bdb8b5801daedccae1ee5434ef1633b03"} Oct 30 18:04:45.187191 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:45.167255 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-738845-6c477d4fb7-r4tnv" event={"ID":"dcb3f0ba-021a-4788-ad64-fcc9054fe5a1","Type":"ContainerDied","Data":"912c16e5d693d332db7d70a406a744ca06e136dbe784f0a8fedd79d124d170e9"} Oct 30 18:04:45.187191 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:45.174461 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb3f0ba-021a-4788-ad64-fcc9054fe5a1-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:45.187191 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:45.174491 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8smb\" (UniqueName: \"kubernetes.io/projected/dcb3f0ba-021a-4788-ad64-fcc9054fe5a1-kube-api-access-w8smb\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:45.202384 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:45.202353 4965 scope.go:122] "RemoveContainer" containerID="c5ab5ce4cb9fe7248cb0a7c171ed70c94540182e0a81bd61540f805ddcff88c2" Oct 30 18:04:45.203573 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:04:45.202849 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5ab5ce4cb9fe7248cb0a7c171ed70c94540182e0a81bd61540f805ddcff88c2\": container with ID starting with c5ab5ce4cb9fe7248cb0a7c171ed70c94540182e0a81bd61540f805ddcff88c2 not found: ID does not exist" containerID="c5ab5ce4cb9fe7248cb0a7c171ed70c94540182e0a81bd61540f805ddcff88c2" Oct 30 18:04:45.203573 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:45.202931 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ab5ce4cb9fe7248cb0a7c171ed70c94540182e0a81bd61540f805ddcff88c2"} err="failed to get container status \"c5ab5ce4cb9fe7248cb0a7c171ed70c94540182e0a81bd61540f805ddcff88c2\": rpc error: code = NotFound desc = could not find container \"c5ab5ce4cb9fe7248cb0a7c171ed70c94540182e0a81bd61540f805ddcff88c2\": container with ID starting with c5ab5ce4cb9fe7248cb0a7c171ed70c94540182e0a81bd61540f805ddcff88c2 not found: ID does not exist" Oct 30 18:04:45.203573 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:45.202957 4965 scope.go:122] "RemoveContainer" containerID="2055b4612290f8e5685b3f98cdaf2f5bdb8b5801daedccae1ee5434ef1633b03" Oct 30 18:04:45.229039 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:45.229008 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-738845-6c477d4fb7-r4tnv"] Oct 30 18:04:45.233379 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:45.233357 4965 scope.go:122] "RemoveContainer" containerID="2055b4612290f8e5685b3f98cdaf2f5bdb8b5801daedccae1ee5434ef1633b03" Oct 30 18:04:45.233748 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:04:45.233722 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2055b4612290f8e5685b3f98cdaf2f5bdb8b5801daedccae1ee5434ef1633b03\": container with ID starting with 2055b4612290f8e5685b3f98cdaf2f5bdb8b5801daedccae1ee5434ef1633b03 not found: ID does not exist" containerID="2055b4612290f8e5685b3f98cdaf2f5bdb8b5801daedccae1ee5434ef1633b03" Oct 30 18:04:45.233851 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:45.233753 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2055b4612290f8e5685b3f98cdaf2f5bdb8b5801daedccae1ee5434ef1633b03"} err="failed to get container status \"2055b4612290f8e5685b3f98cdaf2f5bdb8b5801daedccae1ee5434ef1633b03\": rpc error: code = NotFound desc = could not find container \"2055b4612290f8e5685b3f98cdaf2f5bdb8b5801daedccae1ee5434ef1633b03\": container with ID starting with 2055b4612290f8e5685b3f98cdaf2f5bdb8b5801daedccae1ee5434ef1633b03 not found: ID does not exist" Oct 30 18:04:45.234671 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:45.234650 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-738845-6c477d4fb7-r4tnv"] Oct 30 18:04:45.267833 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:45.267802 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-738845-867b76d77d-qzjng"] Oct 30 18:04:45.272536 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:45.272513 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-738845-867b76d77d-qzjng"] Oct 30 18:04:46.100242 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:46.100212 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8abc8140-8a36-4cc7-80da-43738e049fe0" path="/var/lib/kubelet/pods/8abc8140-8a36-4cc7-80da-43738e049fe0/volumes" Oct 30 18:04:46.100746 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:46.100729 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcb3f0ba-021a-4788-ad64-fcc9054fe5a1" path="/var/lib/kubelet/pods/dcb3f0ba-021a-4788-ad64-fcc9054fe5a1/volumes" Oct 30 18:04:46.739315 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:46.739276 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-01b925-558dd79655-t7zv2"] Oct 30 18:04:51.740619 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:51.740567 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-01b925-558dd79655-t7zv2" podUID="24f2394c-1e9c-417e-91c0-5086ebae37c0" containerName="payload-pre-processing" containerID="cri-o://0a6fea3abed593fc598c1a02f98fd47de968da77b1d59af06db74e66f05237e5" gracePeriod=25 Oct 30 18:04:53.351034 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:53.350998 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj"] Oct 30 18:04:53.353954 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:53.351619 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" podUID="d43a70eb-9850-4990-8385-fd2e2aab5bc2" containerName="istio-proxy" containerID="cri-o://429f4ddfe1d4a6f4a8b8688486d397f86e1536b5c3225f91d069980411be3b72" gracePeriod=30 Oct 30 18:04:53.438127 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:53.438093 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-7458959b4c-fxzzm"] Oct 30 18:04:53.442619 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:53.442578 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-7458959b4c-fxzzm" podUID="d022f31a-8a4f-4a1c-9b99-2b53227e9d49" containerName="payload-pre-processing" containerID="cri-o://5c9c2ca0519c46722b05a346c30c18561e5210e37ce345fd51796e939bd94578" gracePeriod=30 Oct 30 18:04:53.772400 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:53.772373 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-7458959b4c-fxzzm" Oct 30 18:04:53.851929 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:53.848352 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d022f31a-8a4f-4a1c-9b99-2b53227e9d49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d022f31a-8a4f-4a1c-9b99-2b53227e9d49-plugins-config-volume\") pod \"d022f31a-8a4f-4a1c-9b99-2b53227e9d49\" (UID: \"d022f31a-8a4f-4a1c-9b99-2b53227e9d49\") " Oct 30 18:04:53.851929 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:53.848450 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d022f31a-8a4f-4a1c-9b99-2b53227e9d49-kube-api-access-xsb5w\" (UniqueName: \"kubernetes.io/projected/d022f31a-8a4f-4a1c-9b99-2b53227e9d49-kube-api-access-xsb5w\") pod \"d022f31a-8a4f-4a1c-9b99-2b53227e9d49\" (UID: \"d022f31a-8a4f-4a1c-9b99-2b53227e9d49\") " Oct 30 18:04:53.889931 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:53.884116 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d022f31a-8a4f-4a1c-9b99-2b53227e9d49-plugins-config-volume" pod "d022f31a-8a4f-4a1c-9b99-2b53227e9d49" (UID: "d022f31a-8a4f-4a1c-9b99-2b53227e9d49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:04:53.942995 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:53.918100 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d022f31a-8a4f-4a1c-9b99-2b53227e9d49-kube-api-access-xsb5w" pod "d022f31a-8a4f-4a1c-9b99-2b53227e9d49" (UID: "d022f31a-8a4f-4a1c-9b99-2b53227e9d49"). InnerVolumeSpecName "kube-api-access-xsb5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:53.952916 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:53.951541 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d022f31a-8a4f-4a1c-9b99-2b53227e9d49-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:53.952916 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:53.951574 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsb5w\" (UniqueName: \"kubernetes.io/projected/d022f31a-8a4f-4a1c-9b99-2b53227e9d49-kube-api-access-xsb5w\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:54.232912 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:54.232791 4965 generic.go:356] "Generic (PLEG): container finished" podID="d022f31a-8a4f-4a1c-9b99-2b53227e9d49" containerID="5c9c2ca0519c46722b05a346c30c18561e5210e37ce345fd51796e939bd94578" exitCode=0 Oct 30 18:04:54.232912 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:54.232833 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-7458959b4c-fxzzm" Oct 30 18:04:54.232912 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:54.232886 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-7458959b4c-fxzzm" event={"ID":"d022f31a-8a4f-4a1c-9b99-2b53227e9d49","Type":"ContainerDied","Data":"5c9c2ca0519c46722b05a346c30c18561e5210e37ce345fd51796e939bd94578"} Oct 30 18:04:54.233198 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:54.232944 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-7458959b4c-fxzzm" event={"ID":"d022f31a-8a4f-4a1c-9b99-2b53227e9d49","Type":"ContainerDied","Data":"b804d16ddeca58a6f67d322de20930a38a1c9b0bb0b95e6c3e6ac7c40bd633b9"} Oct 30 18:04:54.233198 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:54.232972 4965 scope.go:122] "RemoveContainer" containerID="5c9c2ca0519c46722b05a346c30c18561e5210e37ce345fd51796e939bd94578" Oct 30 18:04:54.243643 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:54.243625 4965 scope.go:122] "RemoveContainer" containerID="5c9c2ca0519c46722b05a346c30c18561e5210e37ce345fd51796e939bd94578" Oct 30 18:04:54.243948 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:04:54.243925 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c9c2ca0519c46722b05a346c30c18561e5210e37ce345fd51796e939bd94578\": container with ID starting with 5c9c2ca0519c46722b05a346c30c18561e5210e37ce345fd51796e939bd94578 not found: ID does not exist" containerID="5c9c2ca0519c46722b05a346c30c18561e5210e37ce345fd51796e939bd94578" Oct 30 18:04:54.244030 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:54.243956 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c9c2ca0519c46722b05a346c30c18561e5210e37ce345fd51796e939bd94578"} err="failed to get container status \"5c9c2ca0519c46722b05a346c30c18561e5210e37ce345fd51796e939bd94578\": rpc error: code = NotFound desc = could not find container \"5c9c2ca0519c46722b05a346c30c18561e5210e37ce345fd51796e939bd94578\": container with ID starting with 5c9c2ca0519c46722b05a346c30c18561e5210e37ce345fd51796e939bd94578 not found: ID does not exist" Oct 30 18:04:54.286016 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:54.285991 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-7458959b4c-fxzzm"] Oct 30 18:04:54.298163 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:54.298140 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b5cbb4-7458959b4c-fxzzm"] Oct 30 18:04:56.100373 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:56.100340 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d022f31a-8a4f-4a1c-9b99-2b53227e9d49" path="/var/lib/kubelet/pods/d022f31a-8a4f-4a1c-9b99-2b53227e9d49/volumes" Oct 30 18:04:58.168852 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:58.168815 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-01b925-558dd79655-t7zv2" podUID="24f2394c-1e9c-417e-91c0-5086ebae37c0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:04:58.650597 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:58.650559 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 18:04:58.720370 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:58.720318 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-credential-socket\") pod \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " Oct 30 18:04:58.720574 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:58.720410 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d43a70eb-9850-4990-8385-fd2e2aab5bc2-kube-api-access-jl99r\" (UniqueName: \"kubernetes.io/projected/d43a70eb-9850-4990-8385-fd2e2aab5bc2-kube-api-access-jl99r\") pod \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " Oct 30 18:04:58.720662 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:58.720597 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istio-podinfo\") pod \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " Oct 30 18:04:58.720662 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:58.720651 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-workload-socket\") pod \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " Oct 30 18:04:58.720822 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:58.720707 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-credential-socket" pod "d43a70eb-9850-4990-8385-fd2e2aab5bc2" (UID: "d43a70eb-9850-4990-8385-fd2e2aab5bc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:58.720822 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:58.720715 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-workload-certs\") pod \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " Oct 30 18:04:58.720822 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:58.720783 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istio-envoy\") pod \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " Oct 30 18:04:58.721019 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:58.720831 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istio-token\" (UniqueName: \"kubernetes.io/projected/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istio-token\") pod \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " Oct 30 18:04:58.721019 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:58.720976 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-workload-certs" pod "d43a70eb-9850-4990-8385-fd2e2aab5bc2" (UID: "d43a70eb-9850-4990-8385-fd2e2aab5bc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:58.721135 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:58.721014 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istiod-ca-cert\") pod \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " Oct 30 18:04:58.721135 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:58.721107 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istio-data\") pod \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\" (UID: \"d43a70eb-9850-4990-8385-fd2e2aab5bc2\") " Oct 30 18:04:58.721278 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:58.721252 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-workload-socket" pod "d43a70eb-9850-4990-8385-fd2e2aab5bc2" (UID: "d43a70eb-9850-4990-8385-fd2e2aab5bc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:58.721414 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:58.721391 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istio-data" pod "d43a70eb-9850-4990-8385-fd2e2aab5bc2" (UID: "d43a70eb-9850-4990-8385-fd2e2aab5bc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:58.721833 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:58.721801 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:58.721833 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:58.721833 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:58.722022 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:58.721847 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:58.722022 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:58.721896 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:58.722192 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:58.722171 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istiod-ca-cert" pod "d43a70eb-9850-4990-8385-fd2e2aab5bc2" (UID: "d43a70eb-9850-4990-8385-fd2e2aab5bc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:04:58.723717 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:58.723652 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istio-envoy" pod "d43a70eb-9850-4990-8385-fd2e2aab5bc2" (UID: "d43a70eb-9850-4990-8385-fd2e2aab5bc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:58.723838 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:58.723785 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d43a70eb-9850-4990-8385-fd2e2aab5bc2-kube-api-access-jl99r" pod "d43a70eb-9850-4990-8385-fd2e2aab5bc2" (UID: "d43a70eb-9850-4990-8385-fd2e2aab5bc2"). InnerVolumeSpecName "kube-api-access-jl99r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:58.723973 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:58.723886 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istio-podinfo" pod "d43a70eb-9850-4990-8385-fd2e2aab5bc2" (UID: "d43a70eb-9850-4990-8385-fd2e2aab5bc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:04:58.724916 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:58.724888 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istio-token" pod "d43a70eb-9850-4990-8385-fd2e2aab5bc2" (UID: "d43a70eb-9850-4990-8385-fd2e2aab5bc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:58.823720 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:58.823683 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl99r\" (UniqueName: \"kubernetes.io/projected/d43a70eb-9850-4990-8385-fd2e2aab5bc2-kube-api-access-jl99r\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:58.823720 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:58.823717 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:58.823720 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:58.823727 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:58.823997 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:58.823736 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:58.823997 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:58.823744 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d43a70eb-9850-4990-8385-fd2e2aab5bc2-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:04:58.928101 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:58.928063 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7"] Oct 30 18:04:59.272420 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:59.272383 4965 generic.go:356] "Generic (PLEG): container finished" podID="d43a70eb-9850-4990-8385-fd2e2aab5bc2" containerID="429f4ddfe1d4a6f4a8b8688486d397f86e1536b5c3225f91d069980411be3b72" exitCode=0 Oct 30 18:04:59.272943 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:59.272434 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" Oct 30 18:04:59.272943 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:59.272460 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" event={"ID":"d43a70eb-9850-4990-8385-fd2e2aab5bc2","Type":"ContainerDied","Data":"429f4ddfe1d4a6f4a8b8688486d397f86e1536b5c3225f91d069980411be3b72"} Oct 30 18:04:59.272943 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:59.272514 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj" event={"ID":"d43a70eb-9850-4990-8385-fd2e2aab5bc2","Type":"ContainerDied","Data":"50f10972378734137f7eeccac67125920083adc8a654e170e7a3540fc646b79b"} Oct 30 18:04:59.272943 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:59.272539 4965 scope.go:122] "RemoveContainer" containerID="429f4ddfe1d4a6f4a8b8688486d397f86e1536b5c3225f91d069980411be3b72" Oct 30 18:04:59.284704 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:59.284684 4965 scope.go:122] "RemoveContainer" containerID="429f4ddfe1d4a6f4a8b8688486d397f86e1536b5c3225f91d069980411be3b72" Oct 30 18:04:59.285081 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:04:59.285036 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"429f4ddfe1d4a6f4a8b8688486d397f86e1536b5c3225f91d069980411be3b72\": container with ID starting with 429f4ddfe1d4a6f4a8b8688486d397f86e1536b5c3225f91d069980411be3b72 not found: ID does not exist" containerID="429f4ddfe1d4a6f4a8b8688486d397f86e1536b5c3225f91d069980411be3b72" Oct 30 18:04:59.285160 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:59.285093 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"429f4ddfe1d4a6f4a8b8688486d397f86e1536b5c3225f91d069980411be3b72"} err="failed to get container status \"429f4ddfe1d4a6f4a8b8688486d397f86e1536b5c3225f91d069980411be3b72\": rpc error: code = NotFound desc = could not find container \"429f4ddfe1d4a6f4a8b8688486d397f86e1536b5c3225f91d069980411be3b72\": container with ID starting with 429f4ddfe1d4a6f4a8b8688486d397f86e1536b5c3225f91d069980411be3b72 not found: ID does not exist" Oct 30 18:04:59.326208 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:59.326180 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj"] Oct 30 18:04:59.340934 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:59.340905 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-hdxhj"] Oct 30 18:04:59.494282 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:59.494253 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fab25f-66d654dd9d-nv9mt"] Oct 30 18:04:59.494472 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:59.494451 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-66d654dd9d-nv9mt" podUID="78b12271-78e7-4b9f-98be-cc3d6b736708" containerName="payload-processing" containerID="cri-o://94de12c06ff2453ff5538008824801c81958cd66240ea926605e8c4459784f56" gracePeriod=30 Oct 30 18:04:59.905544 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:59.896664 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-66d654dd9d-nv9mt" Oct 30 18:04:59.939885 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:59.939796 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78b12271-78e7-4b9f-98be-cc3d6b736708-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78b12271-78e7-4b9f-98be-cc3d6b736708-plugins-config-volume\") pod \"78b12271-78e7-4b9f-98be-cc3d6b736708\" (UID: \"78b12271-78e7-4b9f-98be-cc3d6b736708\") " Oct 30 18:04:59.944880 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:59.940074 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78b12271-78e7-4b9f-98be-cc3d6b736708-kube-api-access-zhv5f\" (UniqueName: \"kubernetes.io/projected/78b12271-78e7-4b9f-98be-cc3d6b736708-kube-api-access-zhv5f\") pod \"78b12271-78e7-4b9f-98be-cc3d6b736708\" (UID: \"78b12271-78e7-4b9f-98be-cc3d6b736708\") " Oct 30 18:04:59.944880 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:59.941244 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78b12271-78e7-4b9f-98be-cc3d6b736708-plugins-config-volume" pod "78b12271-78e7-4b9f-98be-cc3d6b736708" (UID: "78b12271-78e7-4b9f-98be-cc3d6b736708"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:04:59.958855 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:04:59.958822 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b12271-78e7-4b9f-98be-cc3d6b736708-kube-api-access-zhv5f" pod "78b12271-78e7-4b9f-98be-cc3d6b736708" (UID: "78b12271-78e7-4b9f-98be-cc3d6b736708"). InnerVolumeSpecName "kube-api-access-zhv5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:00.051257 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:00.051217 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78b12271-78e7-4b9f-98be-cc3d6b736708-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:00.051257 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:00.051260 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhv5f\" (UniqueName: \"kubernetes.io/projected/78b12271-78e7-4b9f-98be-cc3d6b736708-kube-api-access-zhv5f\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:00.103384 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:00.103353 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d43a70eb-9850-4990-8385-fd2e2aab5bc2" path="/var/lib/kubelet/pods/d43a70eb-9850-4990-8385-fd2e2aab5bc2/volumes" Oct 30 18:05:00.282189 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:00.282108 4965 generic.go:356] "Generic (PLEG): container finished" podID="78b12271-78e7-4b9f-98be-cc3d6b736708" containerID="94de12c06ff2453ff5538008824801c81958cd66240ea926605e8c4459784f56" exitCode=0 Oct 30 18:05:00.282189 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:00.282151 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-66d654dd9d-nv9mt" Oct 30 18:05:00.282725 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:00.282189 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-66d654dd9d-nv9mt" event={"ID":"78b12271-78e7-4b9f-98be-cc3d6b736708","Type":"ContainerDied","Data":"94de12c06ff2453ff5538008824801c81958cd66240ea926605e8c4459784f56"} Oct 30 18:05:00.282725 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:00.282254 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fab25f-66d654dd9d-nv9mt" event={"ID":"78b12271-78e7-4b9f-98be-cc3d6b736708","Type":"ContainerDied","Data":"15709efa76cc2f24554d1a666ec5094c89ce89dd6a8bb25cec14e121c2f2e812"} Oct 30 18:05:00.282725 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:00.282288 4965 scope.go:122] "RemoveContainer" containerID="94de12c06ff2453ff5538008824801c81958cd66240ea926605e8c4459784f56" Oct 30 18:05:00.296770 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:00.296751 4965 scope.go:122] "RemoveContainer" containerID="94de12c06ff2453ff5538008824801c81958cd66240ea926605e8c4459784f56" Oct 30 18:05:00.297130 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:05:00.297104 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94de12c06ff2453ff5538008824801c81958cd66240ea926605e8c4459784f56\": container with ID starting with 94de12c06ff2453ff5538008824801c81958cd66240ea926605e8c4459784f56 not found: ID does not exist" containerID="94de12c06ff2453ff5538008824801c81958cd66240ea926605e8c4459784f56" Oct 30 18:05:00.297225 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:00.297137 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94de12c06ff2453ff5538008824801c81958cd66240ea926605e8c4459784f56"} err="failed to get container status \"94de12c06ff2453ff5538008824801c81958cd66240ea926605e8c4459784f56\": rpc error: code = NotFound desc = could not find container \"94de12c06ff2453ff5538008824801c81958cd66240ea926605e8c4459784f56\": container with ID starting with 94de12c06ff2453ff5538008824801c81958cd66240ea926605e8c4459784f56 not found: ID does not exist" Oct 30 18:05:00.335245 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:00.335213 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fab25f-66d654dd9d-nv9mt"] Oct 30 18:05:00.348871 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:00.348824 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fab25f-66d654dd9d-nv9mt"] Oct 30 18:05:02.144887 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:02.144737 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78b12271-78e7-4b9f-98be-cc3d6b736708" path="/var/lib/kubelet/pods/78b12271-78e7-4b9f-98be-cc3d6b736708/volumes" Oct 30 18:05:05.997136 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:05:05.997102 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:05:07.370357 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:07.370326 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd"] Oct 30 18:05:07.370982 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:07.370576 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" podUID="241c1434-3e5c-4120-8eb5-507bf30a92ec" containerName="istio-proxy" containerID="cri-o://da12ea0900c7ab9fa9715f6656827db1d7b795e28eb111c4f0bbfb3b7b3d5b03" gracePeriod=30 Oct 30 18:05:08.169560 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:08.169524 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-01b925-558dd79655-t7zv2" podUID="24f2394c-1e9c-417e-91c0-5086ebae37c0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:05:11.032997 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:11.032961 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-785eb5bf-tgr2l"] Oct 30 18:05:11.045692 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:11.045659 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-785eb5bf-tgr2l"] Oct 30 18:05:12.073358 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.073331 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-01b925-558dd79655-t7zv2" Oct 30 18:05:12.101631 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.101596 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43ab0275-09f7-45c9-bc2b-4e9893421b38" path="/var/lib/kubelet/pods/43ab0275-09f7-45c9-bc2b-4e9893421b38/volumes" Oct 30 18:05:12.250426 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.250332 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24f2394c-1e9c-417e-91c0-5086ebae37c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24f2394c-1e9c-417e-91c0-5086ebae37c0-plugins-config-volume\") pod \"24f2394c-1e9c-417e-91c0-5086ebae37c0\" (UID: \"24f2394c-1e9c-417e-91c0-5086ebae37c0\") " Oct 30 18:05:12.250629 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.250428 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24f2394c-1e9c-417e-91c0-5086ebae37c0-kube-api-access-6hnz6\" (UniqueName: \"kubernetes.io/projected/24f2394c-1e9c-417e-91c0-5086ebae37c0-kube-api-access-6hnz6\") pod \"24f2394c-1e9c-417e-91c0-5086ebae37c0\" (UID: \"24f2394c-1e9c-417e-91c0-5086ebae37c0\") " Oct 30 18:05:12.250799 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.250765 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24f2394c-1e9c-417e-91c0-5086ebae37c0-plugins-config-volume" pod "24f2394c-1e9c-417e-91c0-5086ebae37c0" (UID: "24f2394c-1e9c-417e-91c0-5086ebae37c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:05:12.251602 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.251579 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24f2394c-1e9c-417e-91c0-5086ebae37c0-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:12.253172 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.253142 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f2394c-1e9c-417e-91c0-5086ebae37c0-kube-api-access-6hnz6" pod "24f2394c-1e9c-417e-91c0-5086ebae37c0" (UID: "24f2394c-1e9c-417e-91c0-5086ebae37c0"). InnerVolumeSpecName "kube-api-access-6hnz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:12.352777 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.352745 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hnz6\" (UniqueName: \"kubernetes.io/projected/24f2394c-1e9c-417e-91c0-5086ebae37c0-kube-api-access-6hnz6\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:12.431893 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.431854 4965 generic.go:356] "Generic (PLEG): container finished" podID="24f2394c-1e9c-417e-91c0-5086ebae37c0" containerID="0a6fea3abed593fc598c1a02f98fd47de968da77b1d59af06db74e66f05237e5" exitCode=0 Oct 30 18:05:12.432027 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.431908 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-01b925-558dd79655-t7zv2" Oct 30 18:05:12.432027 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.431914 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-01b925-558dd79655-t7zv2" event={"ID":"24f2394c-1e9c-417e-91c0-5086ebae37c0","Type":"ContainerDied","Data":"0a6fea3abed593fc598c1a02f98fd47de968da77b1d59af06db74e66f05237e5"} Oct 30 18:05:12.432027 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.431981 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-01b925-558dd79655-t7zv2" event={"ID":"24f2394c-1e9c-417e-91c0-5086ebae37c0","Type":"ContainerDied","Data":"0a63395915398e913326262ec8b96559ff4c03300d59a2e3a5ba41df997d71f5"} Oct 30 18:05:12.432027 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.432012 4965 scope.go:122] "RemoveContainer" containerID="0a6fea3abed593fc598c1a02f98fd47de968da77b1d59af06db74e66f05237e5" Oct 30 18:05:12.484274 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.484247 4965 scope.go:122] "RemoveContainer" containerID="0a6fea3abed593fc598c1a02f98fd47de968da77b1d59af06db74e66f05237e5" Oct 30 18:05:12.484600 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:05:12.484568 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a6fea3abed593fc598c1a02f98fd47de968da77b1d59af06db74e66f05237e5\": container with ID starting with 0a6fea3abed593fc598c1a02f98fd47de968da77b1d59af06db74e66f05237e5 not found: ID does not exist" containerID="0a6fea3abed593fc598c1a02f98fd47de968da77b1d59af06db74e66f05237e5" Oct 30 18:05:12.484675 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.484608 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a6fea3abed593fc598c1a02f98fd47de968da77b1d59af06db74e66f05237e5"} err="failed to get container status \"0a6fea3abed593fc598c1a02f98fd47de968da77b1d59af06db74e66f05237e5\": rpc error: code = NotFound desc = could not find container \"0a6fea3abed593fc598c1a02f98fd47de968da77b1d59af06db74e66f05237e5\": container with ID starting with 0a6fea3abed593fc598c1a02f98fd47de968da77b1d59af06db74e66f05237e5 not found: ID does not exist" Oct 30 18:05:12.499644 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.499619 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-01b925-558dd79655-t7zv2"] Oct 30 18:05:12.511503 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.511449 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-01b925-558dd79655-t7zv2"] Oct 30 18:05:12.635601 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.635574 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 18:05:12.757987 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.757953 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/241c1434-3e5c-4120-8eb5-507bf30a92ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/241c1434-3e5c-4120-8eb5-507bf30a92ec-istiod-ca-cert\") pod \"241c1434-3e5c-4120-8eb5-507bf30a92ec\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " Oct 30 18:05:12.758186 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.758148 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-workload-socket\") pod \"241c1434-3e5c-4120-8eb5-507bf30a92ec\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " Oct 30 18:05:12.758276 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.758191 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/241c1434-3e5c-4120-8eb5-507bf30a92ec-kube-api-access-xcgfl\" (UniqueName: \"kubernetes.io/projected/241c1434-3e5c-4120-8eb5-507bf30a92ec-kube-api-access-xcgfl\") pod \"241c1434-3e5c-4120-8eb5-507bf30a92ec\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " Oct 30 18:05:12.758276 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.758229 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-istio-envoy\") pod \"241c1434-3e5c-4120-8eb5-507bf30a92ec\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " Oct 30 18:05:12.758391 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.758276 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-workload-certs\") pod \"241c1434-3e5c-4120-8eb5-507bf30a92ec\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " Oct 30 18:05:12.758453 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.758427 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/241c1434-3e5c-4120-8eb5-507bf30a92ec-istio-token\" (UniqueName: \"kubernetes.io/projected/241c1434-3e5c-4120-8eb5-507bf30a92ec-istio-token\") pod \"241c1434-3e5c-4120-8eb5-507bf30a92ec\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " Oct 30 18:05:12.758513 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.758489 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-istio-data\") pod \"241c1434-3e5c-4120-8eb5-507bf30a92ec\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " Oct 30 18:05:12.758569 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.758520 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-credential-socket\") pod \"241c1434-3e5c-4120-8eb5-507bf30a92ec\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " Oct 30 18:05:12.758624 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.758581 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/241c1434-3e5c-4120-8eb5-507bf30a92ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/241c1434-3e5c-4120-8eb5-507bf30a92ec-istio-podinfo\") pod \"241c1434-3e5c-4120-8eb5-507bf30a92ec\" (UID: \"241c1434-3e5c-4120-8eb5-507bf30a92ec\") " Oct 30 18:05:12.759795 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.759332 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-workload-certs" pod "241c1434-3e5c-4120-8eb5-507bf30a92ec" (UID: "241c1434-3e5c-4120-8eb5-507bf30a92ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:12.759795 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.759356 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-istio-data" pod "241c1434-3e5c-4120-8eb5-507bf30a92ec" (UID: "241c1434-3e5c-4120-8eb5-507bf30a92ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:12.759795 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.759582 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/241c1434-3e5c-4120-8eb5-507bf30a92ec-istiod-ca-cert" pod "241c1434-3e5c-4120-8eb5-507bf30a92ec" (UID: "241c1434-3e5c-4120-8eb5-507bf30a92ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:05:12.759795 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.759665 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-credential-socket" pod "241c1434-3e5c-4120-8eb5-507bf30a92ec" (UID: "241c1434-3e5c-4120-8eb5-507bf30a92ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:12.759795 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.759734 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/241c1434-3e5c-4120-8eb5-507bf30a92ec-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:12.759795 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.759754 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:12.759795 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.759767 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:12.760581 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.760558 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-workload-socket" pod "241c1434-3e5c-4120-8eb5-507bf30a92ec" (UID: "241c1434-3e5c-4120-8eb5-507bf30a92ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:12.763314 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.763244 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/241c1434-3e5c-4120-8eb5-507bf30a92ec-istio-podinfo" pod "241c1434-3e5c-4120-8eb5-507bf30a92ec" (UID: "241c1434-3e5c-4120-8eb5-507bf30a92ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:05:12.763581 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.763554 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-istio-envoy" pod "241c1434-3e5c-4120-8eb5-507bf30a92ec" (UID: "241c1434-3e5c-4120-8eb5-507bf30a92ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:12.764520 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.764497 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/241c1434-3e5c-4120-8eb5-507bf30a92ec-istio-token" pod "241c1434-3e5c-4120-8eb5-507bf30a92ec" (UID: "241c1434-3e5c-4120-8eb5-507bf30a92ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:12.764606 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.764536 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/241c1434-3e5c-4120-8eb5-507bf30a92ec-kube-api-access-xcgfl" pod "241c1434-3e5c-4120-8eb5-507bf30a92ec" (UID: "241c1434-3e5c-4120-8eb5-507bf30a92ec"). InnerVolumeSpecName "kube-api-access-xcgfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:12.861608 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.861581 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/241c1434-3e5c-4120-8eb5-507bf30a92ec-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:12.861608 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.861607 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:12.861764 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.861618 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/241c1434-3e5c-4120-8eb5-507bf30a92ec-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:12.861764 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.861628 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:12.861764 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.861637 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcgfl\" (UniqueName: \"kubernetes.io/projected/241c1434-3e5c-4120-8eb5-507bf30a92ec-kube-api-access-xcgfl\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:12.861764 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:12.861646 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/241c1434-3e5c-4120-8eb5-507bf30a92ec-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:13.111669 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:13.111635 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a17354-575588998c-57bcf"] Oct 30 18:05:13.441896 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:13.441774 4965 generic.go:356] "Generic (PLEG): container finished" podID="241c1434-3e5c-4120-8eb5-507bf30a92ec" containerID="da12ea0900c7ab9fa9715f6656827db1d7b795e28eb111c4f0bbfb3b7b3d5b03" exitCode=0 Oct 30 18:05:13.441896 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:13.441830 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" Oct 30 18:05:13.442146 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:13.441855 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" event={"ID":"241c1434-3e5c-4120-8eb5-507bf30a92ec","Type":"ContainerDied","Data":"da12ea0900c7ab9fa9715f6656827db1d7b795e28eb111c4f0bbfb3b7b3d5b03"} Oct 30 18:05:13.442146 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:13.441953 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd" event={"ID":"241c1434-3e5c-4120-8eb5-507bf30a92ec","Type":"ContainerDied","Data":"ba544a613cdf99bce0e87e927ea0342e6e75f3d92688222727723fdf9c776ff1"} Oct 30 18:05:13.442146 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:13.441992 4965 scope.go:122] "RemoveContainer" containerID="da12ea0900c7ab9fa9715f6656827db1d7b795e28eb111c4f0bbfb3b7b3d5b03" Oct 30 18:05:13.456992 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:13.456788 4965 scope.go:122] "RemoveContainer" containerID="da12ea0900c7ab9fa9715f6656827db1d7b795e28eb111c4f0bbfb3b7b3d5b03" Oct 30 18:05:13.457303 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:05:13.457227 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da12ea0900c7ab9fa9715f6656827db1d7b795e28eb111c4f0bbfb3b7b3d5b03\": container with ID starting with da12ea0900c7ab9fa9715f6656827db1d7b795e28eb111c4f0bbfb3b7b3d5b03 not found: ID does not exist" containerID="da12ea0900c7ab9fa9715f6656827db1d7b795e28eb111c4f0bbfb3b7b3d5b03" Oct 30 18:05:13.457303 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:13.457271 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da12ea0900c7ab9fa9715f6656827db1d7b795e28eb111c4f0bbfb3b7b3d5b03"} err="failed to get container status \"da12ea0900c7ab9fa9715f6656827db1d7b795e28eb111c4f0bbfb3b7b3d5b03\": rpc error: code = NotFound desc = could not find container \"da12ea0900c7ab9fa9715f6656827db1d7b795e28eb111c4f0bbfb3b7b3d5b03\": container with ID starting with da12ea0900c7ab9fa9715f6656827db1d7b795e28eb111c4f0bbfb3b7b3d5b03 not found: ID does not exist" Oct 30 18:05:13.498060 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:13.498029 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd"] Oct 30 18:05:13.509938 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:13.509907 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-r9kqd"] Oct 30 18:05:13.953565 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:13.953515 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" podUID="d174a74f-7205-4020-94b0-e30652c9b721" containerName="main" containerID="cri-o://e0ce0b77cde220bc1f335db9d3713db548d8e67d493f6889a0bf14b160c14ca2" gracePeriod=45 Oct 30 18:05:14.107076 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.107045 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="241c1434-3e5c-4120-8eb5-507bf30a92ec" path="/var/lib/kubelet/pods/241c1434-3e5c-4120-8eb5-507bf30a92ec/volumes" Oct 30 18:05:14.107490 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.107473 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24f2394c-1e9c-417e-91c0-5086ebae37c0" path="/var/lib/kubelet/pods/24f2394c-1e9c-417e-91c0-5086ebae37c0/volumes" Oct 30 18:05:14.135379 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.135344 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9"] Oct 30 18:05:14.135910 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.135534 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9" podUID="40110c67-890b-40ea-ad4c-0c673666f1cd" containerName="maas-api" containerID="cri-o://c5f43f964ad914c9d47741e9e207aab7d5ffea95efad6766e1cf06d4c1fdb72c" gracePeriod=30 Oct 30 18:05:14.314116 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.314090 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" Oct 30 18:05:14.457594 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.457561 4965 generic.go:356] "Generic (PLEG): container finished" podID="d174a74f-7205-4020-94b0-e30652c9b721" containerID="e0ce0b77cde220bc1f335db9d3713db548d8e67d493f6889a0bf14b160c14ca2" exitCode=0 Oct 30 18:05:14.457796 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.457716 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" event={"ID":"d174a74f-7205-4020-94b0-e30652c9b721","Type":"ContainerDied","Data":"e0ce0b77cde220bc1f335db9d3713db548d8e67d493f6889a0bf14b160c14ca2"} Oct 30 18:05:14.457796 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.457753 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" event={"ID":"d174a74f-7205-4020-94b0-e30652c9b721","Type":"ContainerDied","Data":"535ac247b14dc551828580b4d956236f623ba1caf15c11fc092cf71970df265f"} Oct 30 18:05:14.457796 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.457793 4965 scope.go:122] "RemoveContainer" containerID="e0ce0b77cde220bc1f335db9d3713db548d8e67d493f6889a0bf14b160c14ca2" Oct 30 18:05:14.458004 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.457952 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7" Oct 30 18:05:14.461463 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.461435 4965 generic.go:356] "Generic (PLEG): container finished" podID="40110c67-890b-40ea-ad4c-0c673666f1cd" containerID="c5f43f964ad914c9d47741e9e207aab7d5ffea95efad6766e1cf06d4c1fdb72c" exitCode=0 Oct 30 18:05:14.461597 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.461490 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9" event={"ID":"40110c67-890b-40ea-ad4c-0c673666f1cd","Type":"ContainerDied","Data":"c5f43f964ad914c9d47741e9e207aab7d5ffea95efad6766e1cf06d4c1fdb72c"} Oct 30 18:05:14.461597 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.461539 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9" event={"ID":"40110c67-890b-40ea-ad4c-0c673666f1cd","Type":"ContainerDied","Data":"433227b4642c2bb5798af760062f8711cc236af8a2752f69e21ef588c7224ce0"} Oct 30 18:05:14.461597 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.461566 4965 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="433227b4642c2bb5798af760062f8711cc236af8a2752f69e21ef588c7224ce0" Oct 30 18:05:14.464346 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.464319 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9" Oct 30 18:05:14.470630 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.470593 4965 scope.go:122] "RemoveContainer" containerID="e0ce0b77cde220bc1f335db9d3713db548d8e67d493f6889a0bf14b160c14ca2" Oct 30 18:05:14.470932 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:05:14.470905 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0ce0b77cde220bc1f335db9d3713db548d8e67d493f6889a0bf14b160c14ca2\": container with ID starting with e0ce0b77cde220bc1f335db9d3713db548d8e67d493f6889a0bf14b160c14ca2 not found: ID does not exist" containerID="e0ce0b77cde220bc1f335db9d3713db548d8e67d493f6889a0bf14b160c14ca2" Oct 30 18:05:14.471024 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.470946 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0ce0b77cde220bc1f335db9d3713db548d8e67d493f6889a0bf14b160c14ca2"} err="failed to get container status \"e0ce0b77cde220bc1f335db9d3713db548d8e67d493f6889a0bf14b160c14ca2\": rpc error: code = NotFound desc = could not find container \"e0ce0b77cde220bc1f335db9d3713db548d8e67d493f6889a0bf14b160c14ca2\": container with ID starting with e0ce0b77cde220bc1f335db9d3713db548d8e67d493f6889a0bf14b160c14ca2 not found: ID does not exist" Oct 30 18:05:14.493789 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.493753 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d174a74f-7205-4020-94b0-e30652c9b721-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d174a74f-7205-4020-94b0-e30652c9b721-tmp-dir\") pod \"d174a74f-7205-4020-94b0-e30652c9b721\" (UID: \"d174a74f-7205-4020-94b0-e30652c9b721\") " Oct 30 18:05:14.493917 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.493871 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d174a74f-7205-4020-94b0-e30652c9b721-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d174a74f-7205-4020-94b0-e30652c9b721-dshm\") pod \"d174a74f-7205-4020-94b0-e30652c9b721\" (UID: \"d174a74f-7205-4020-94b0-e30652c9b721\") " Oct 30 18:05:14.493960 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.493925 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d174a74f-7205-4020-94b0-e30652c9b721-tls-certs\" (UniqueName: \"kubernetes.io/secret/d174a74f-7205-4020-94b0-e30652c9b721-tls-certs\") pod \"d174a74f-7205-4020-94b0-e30652c9b721\" (UID: \"d174a74f-7205-4020-94b0-e30652c9b721\") " Oct 30 18:05:14.494004 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.493958 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d174a74f-7205-4020-94b0-e30652c9b721-home\" (UniqueName: \"kubernetes.io/empty-dir/d174a74f-7205-4020-94b0-e30652c9b721-home\") pod \"d174a74f-7205-4020-94b0-e30652c9b721\" (UID: \"d174a74f-7205-4020-94b0-e30652c9b721\") " Oct 30 18:05:14.494048 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.494004 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d174a74f-7205-4020-94b0-e30652c9b721-kube-api-access-w4d7p\" (UniqueName: \"kubernetes.io/projected/d174a74f-7205-4020-94b0-e30652c9b721-kube-api-access-w4d7p\") pod \"d174a74f-7205-4020-94b0-e30652c9b721\" (UID: \"d174a74f-7205-4020-94b0-e30652c9b721\") " Oct 30 18:05:14.494093 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.494053 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d174a74f-7205-4020-94b0-e30652c9b721-tmp-dir" pod "d174a74f-7205-4020-94b0-e30652c9b721" (UID: "d174a74f-7205-4020-94b0-e30652c9b721"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:14.494230 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.494207 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d174a74f-7205-4020-94b0-e30652c9b721-home" pod "d174a74f-7205-4020-94b0-e30652c9b721" (UID: "d174a74f-7205-4020-94b0-e30652c9b721"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:14.494295 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.494228 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d174a74f-7205-4020-94b0-e30652c9b721-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d174a74f-7205-4020-94b0-e30652c9b721-model-cache\") pod \"d174a74f-7205-4020-94b0-e30652c9b721\" (UID: \"d174a74f-7205-4020-94b0-e30652c9b721\") " Oct 30 18:05:14.494516 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.494480 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d174a74f-7205-4020-94b0-e30652c9b721-model-cache" pod "d174a74f-7205-4020-94b0-e30652c9b721" (UID: "d174a74f-7205-4020-94b0-e30652c9b721"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:14.494853 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.494834 4965 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d174a74f-7205-4020-94b0-e30652c9b721-tmp-dir\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:14.494953 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.494899 4965 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d174a74f-7205-4020-94b0-e30652c9b721-home\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:14.494953 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.494915 4965 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d174a74f-7205-4020-94b0-e30652c9b721-model-cache\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:14.496077 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.496057 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d174a74f-7205-4020-94b0-e30652c9b721-dshm" pod "d174a74f-7205-4020-94b0-e30652c9b721" (UID: "d174a74f-7205-4020-94b0-e30652c9b721"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:14.496141 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.496091 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d174a74f-7205-4020-94b0-e30652c9b721-tls-certs" pod "d174a74f-7205-4020-94b0-e30652c9b721" (UID: "d174a74f-7205-4020-94b0-e30652c9b721"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:05:14.496202 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.496185 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d174a74f-7205-4020-94b0-e30652c9b721-kube-api-access-w4d7p" pod "d174a74f-7205-4020-94b0-e30652c9b721" (UID: "d174a74f-7205-4020-94b0-e30652c9b721"). InnerVolumeSpecName "kube-api-access-w4d7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:14.595537 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.595495 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40110c67-890b-40ea-ad4c-0c673666f1cd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40110c67-890b-40ea-ad4c-0c673666f1cd-maas-api-tls\") pod \"40110c67-890b-40ea-ad4c-0c673666f1cd\" (UID: \"40110c67-890b-40ea-ad4c-0c673666f1cd\") " Oct 30 18:05:14.595718 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.595612 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40110c67-890b-40ea-ad4c-0c673666f1cd-kube-api-access-9r9ss\" (UniqueName: \"kubernetes.io/projected/40110c67-890b-40ea-ad4c-0c673666f1cd-kube-api-access-9r9ss\") pod \"40110c67-890b-40ea-ad4c-0c673666f1cd\" (UID: \"40110c67-890b-40ea-ad4c-0c673666f1cd\") " Oct 30 18:05:14.595846 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.595826 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40110c67-890b-40ea-ad4c-0c673666f1cd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40110c67-890b-40ea-ad4c-0c673666f1cd-openshift-service-ca\") pod \"40110c67-890b-40ea-ad4c-0c673666f1cd\" (UID: \"40110c67-890b-40ea-ad4c-0c673666f1cd\") " Oct 30 18:05:14.595941 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.595912 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40110c67-890b-40ea-ad4c-0c673666f1cd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/40110c67-890b-40ea-ad4c-0c673666f1cd-metrics-certs\") pod \"40110c67-890b-40ea-ad4c-0c673666f1cd\" (UID: \"40110c67-890b-40ea-ad4c-0c673666f1cd\") " Oct 30 18:05:14.596248 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.596216 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40110c67-890b-40ea-ad4c-0c673666f1cd-openshift-service-ca" pod "40110c67-890b-40ea-ad4c-0c673666f1cd" (UID: "40110c67-890b-40ea-ad4c-0c673666f1cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:05:14.596620 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.596594 4965 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40110c67-890b-40ea-ad4c-0c673666f1cd-openshift-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:14.596717 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.596624 4965 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d174a74f-7205-4020-94b0-e30652c9b721-dshm\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:14.596717 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.596640 4965 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d174a74f-7205-4020-94b0-e30652c9b721-tls-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:14.596717 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.596657 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4d7p\" (UniqueName: \"kubernetes.io/projected/d174a74f-7205-4020-94b0-e30652c9b721-kube-api-access-w4d7p\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:14.597759 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.597735 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40110c67-890b-40ea-ad4c-0c673666f1cd-maas-api-tls" pod "40110c67-890b-40ea-ad4c-0c673666f1cd" (UID: "40110c67-890b-40ea-ad4c-0c673666f1cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:05:14.597852 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.597788 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40110c67-890b-40ea-ad4c-0c673666f1cd-kube-api-access-9r9ss" pod "40110c67-890b-40ea-ad4c-0c673666f1cd" (UID: "40110c67-890b-40ea-ad4c-0c673666f1cd"). InnerVolumeSpecName "kube-api-access-9r9ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:14.598010 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.597951 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40110c67-890b-40ea-ad4c-0c673666f1cd-metrics-certs" pod "40110c67-890b-40ea-ad4c-0c673666f1cd" (UID: "40110c67-890b-40ea-ad4c-0c673666f1cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:05:14.715086 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.700058 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40110c67-890b-40ea-ad4c-0c673666f1cd-maas-api-tls\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:14.715086 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.700094 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r9ss\" (UniqueName: \"kubernetes.io/projected/40110c67-890b-40ea-ad4c-0c673666f1cd-kube-api-access-9r9ss\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:14.715086 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.700109 4965 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40110c67-890b-40ea-ad4c-0c673666f1cd-metrics-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:14.840550 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.840403 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7"] Oct 30 18:05:14.847514 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:14.847445 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-306aed/distinct-w3-306aed-kserve-64b5b4fbd5-bqlq7"] Oct 30 18:05:15.470129 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:15.470098 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9" Oct 30 18:05:15.523119 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:15.523081 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9"] Oct 30 18:05:15.534794 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:15.534765 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-23c1e8-78897c98f4-5r9x9"] Oct 30 18:05:16.099958 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:16.099924 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40110c67-890b-40ea-ad4c-0c673666f1cd" path="/var/lib/kubelet/pods/40110c67-890b-40ea-ad4c-0c673666f1cd/volumes" Oct 30 18:05:16.100370 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:16.100355 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d174a74f-7205-4020-94b0-e30652c9b721" path="/var/lib/kubelet/pods/d174a74f-7205-4020-94b0-e30652c9b721/volumes" Oct 30 18:05:18.112018 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:18.111975 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a17354-575588998c-57bcf" podUID="404cc4e8-d6c9-4d5b-8abe-c68527da5248" containerName="payload-pre-processing" containerID="cri-o://266daea88c68d2ea47ecb788f6fb4c8f5969da95bc710c48e85ead45997ed373" gracePeriod=25 Oct 30 18:05:18.435551 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:18.435519 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a17354-575588998c-57bcf" Oct 30 18:05:18.495006 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:18.494967 4965 generic.go:356] "Generic (PLEG): container finished" podID="404cc4e8-d6c9-4d5b-8abe-c68527da5248" containerID="266daea88c68d2ea47ecb788f6fb4c8f5969da95bc710c48e85ead45997ed373" exitCode=0 Oct 30 18:05:18.495211 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:18.495008 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a17354-575588998c-57bcf" Oct 30 18:05:18.495211 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:18.495050 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a17354-575588998c-57bcf" event={"ID":"404cc4e8-d6c9-4d5b-8abe-c68527da5248","Type":"ContainerDied","Data":"266daea88c68d2ea47ecb788f6fb4c8f5969da95bc710c48e85ead45997ed373"} Oct 30 18:05:18.495211 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:18.495115 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a17354-575588998c-57bcf" event={"ID":"404cc4e8-d6c9-4d5b-8abe-c68527da5248","Type":"ContainerDied","Data":"21b1de6fbc5dad5450a029769dafbcd5b5d8e28b1ee6f3d982eb7da2740ba1e5"} Oct 30 18:05:18.495211 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:18.495145 4965 scope.go:122] "RemoveContainer" containerID="266daea88c68d2ea47ecb788f6fb4c8f5969da95bc710c48e85ead45997ed373" Oct 30 18:05:18.510696 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:18.510676 4965 scope.go:122] "RemoveContainer" containerID="266daea88c68d2ea47ecb788f6fb4c8f5969da95bc710c48e85ead45997ed373" Oct 30 18:05:18.511038 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:05:18.511011 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"266daea88c68d2ea47ecb788f6fb4c8f5969da95bc710c48e85ead45997ed373\": container with ID starting with 266daea88c68d2ea47ecb788f6fb4c8f5969da95bc710c48e85ead45997ed373 not found: ID does not exist" containerID="266daea88c68d2ea47ecb788f6fb4c8f5969da95bc710c48e85ead45997ed373" Oct 30 18:05:18.511128 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:18.511049 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"266daea88c68d2ea47ecb788f6fb4c8f5969da95bc710c48e85ead45997ed373"} err="failed to get container status \"266daea88c68d2ea47ecb788f6fb4c8f5969da95bc710c48e85ead45997ed373\": rpc error: code = NotFound desc = could not find container \"266daea88c68d2ea47ecb788f6fb4c8f5969da95bc710c48e85ead45997ed373\": container with ID starting with 266daea88c68d2ea47ecb788f6fb4c8f5969da95bc710c48e85ead45997ed373 not found: ID does not exist" Oct 30 18:05:18.572294 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:18.572267 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/404cc4e8-d6c9-4d5b-8abe-c68527da5248-kube-api-access-4fqh7\" (UniqueName: \"kubernetes.io/projected/404cc4e8-d6c9-4d5b-8abe-c68527da5248-kube-api-access-4fqh7\") pod \"404cc4e8-d6c9-4d5b-8abe-c68527da5248\" (UID: \"404cc4e8-d6c9-4d5b-8abe-c68527da5248\") " Oct 30 18:05:18.572428 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:18.572403 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/404cc4e8-d6c9-4d5b-8abe-c68527da5248-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/404cc4e8-d6c9-4d5b-8abe-c68527da5248-plugins-config-volume\") pod \"404cc4e8-d6c9-4d5b-8abe-c68527da5248\" (UID: \"404cc4e8-d6c9-4d5b-8abe-c68527da5248\") " Oct 30 18:05:18.572747 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:18.572725 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/404cc4e8-d6c9-4d5b-8abe-c68527da5248-plugins-config-volume" pod "404cc4e8-d6c9-4d5b-8abe-c68527da5248" (UID: "404cc4e8-d6c9-4d5b-8abe-c68527da5248"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:05:18.572964 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:18.572949 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/404cc4e8-d6c9-4d5b-8abe-c68527da5248-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:18.574422 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:18.574398 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/404cc4e8-d6c9-4d5b-8abe-c68527da5248-kube-api-access-4fqh7" pod "404cc4e8-d6c9-4d5b-8abe-c68527da5248" (UID: "404cc4e8-d6c9-4d5b-8abe-c68527da5248"). InnerVolumeSpecName "kube-api-access-4fqh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:18.674462 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:18.674389 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fqh7\" (UniqueName: \"kubernetes.io/projected/404cc4e8-d6c9-4d5b-8abe-c68527da5248-kube-api-access-4fqh7\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:18.841682 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:18.841650 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a17354-575588998c-57bcf"] Oct 30 18:05:18.854171 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:18.854138 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a17354-575588998c-57bcf"] Oct 30 18:05:20.099446 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:20.099411 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="404cc4e8-d6c9-4d5b-8abe-c68527da5248" path="/var/lib/kubelet/pods/404cc4e8-d6c9-4d5b-8abe-c68527da5248/volumes" Oct 30 18:05:22.030818 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:22.030787 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b52a3f74-l9swh"] Oct 30 18:05:22.043036 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:22.043008 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b52a3f74-l9swh"] Oct 30 18:05:22.101339 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:22.101310 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e43b107-9066-4368-a04a-8437c7b4201d" path="/var/lib/kubelet/pods/5e43b107-9066-4368-a04a-8437c7b4201d/volumes" Oct 30 18:05:22.657010 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:22.656975 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv"] Oct 30 18:05:22.657234 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:22.657205 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" podUID="722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b" containerName="istio-proxy" containerID="cri-o://55919ca08474befb4c087b7bd0d75e85118eb591dd5755982eb42a27ea686318" gracePeriod=30 Oct 30 18:05:24.356502 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:24.356470 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:05:26.199750 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:26.199711 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:05:26.200174 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:26.199790 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" podUID="722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:05:26.834907 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:26.834874 4965 scope.go:122] "RemoveContainer" containerID="c5f43f964ad914c9d47741e9e207aab7d5ffea95efad6766e1cf06d4c1fdb72c" Oct 30 18:05:27.943841 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:27.943782 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 18:05:28.065930 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.065896 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-credential-socket\") pod \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " Oct 30 18:05:28.066156 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.065953 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istiod-ca-cert\") pod \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " Oct 30 18:05:28.066156 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.065978 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istio-podinfo\") pod \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " Oct 30 18:05:28.066156 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.066029 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istio-envoy\") pod \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " Oct 30 18:05:28.066156 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.066087 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istio-token\" (UniqueName: \"kubernetes.io/projected/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istio-token\") pod \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " Oct 30 18:05:28.066156 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.066116 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-credential-socket" pod "722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b" (UID: "722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:28.066492 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.066166 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istio-data\") pod \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " Oct 30 18:05:28.066492 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.066325 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-workload-certs\") pod \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " Oct 30 18:05:28.066492 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.066383 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istiod-ca-cert" pod "722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b" (UID: "722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:05:28.066492 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.066394 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-kube-api-access-f4fp5\" (UniqueName: \"kubernetes.io/projected/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-kube-api-access-f4fp5\") pod \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " Oct 30 18:05:28.066492 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.066462 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-workload-socket\") pod \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\" (UID: \"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b\") " Oct 30 18:05:28.066771 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.066545 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-workload-certs" pod "722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b" (UID: "722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:28.066827 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.066802 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istio-data" pod "722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b" (UID: "722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:28.066927 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.066857 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-workload-socket" pod "722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b" (UID: "722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:28.067233 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.067209 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:28.067326 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.067239 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:28.067326 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.067255 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:28.067326 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.067269 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:28.067326 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.067284 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:28.068547 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.068520 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-kube-api-access-f4fp5" pod "722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b" (UID: "722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b"). InnerVolumeSpecName "kube-api-access-f4fp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:28.068636 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.068563 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istio-token" pod "722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b" (UID: "722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:28.068803 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.068781 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istio-podinfo" pod "722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b" (UID: "722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:05:28.069019 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.068996 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istio-envoy" pod "722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b" (UID: "722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:28.168748 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.168717 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:28.168748 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.168744 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:28.168748 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.168753 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:28.169000 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.168762 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4fp5\" (UniqueName: \"kubernetes.io/projected/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b-kube-api-access-f4fp5\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:28.580742 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.580707 4965 generic.go:356] "Generic (PLEG): container finished" podID="722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b" containerID="55919ca08474befb4c087b7bd0d75e85118eb591dd5755982eb42a27ea686318" exitCode=0 Oct 30 18:05:28.580982 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.580790 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" Oct 30 18:05:28.580982 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.580804 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" event={"ID":"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b","Type":"ContainerDied","Data":"55919ca08474befb4c087b7bd0d75e85118eb591dd5755982eb42a27ea686318"} Oct 30 18:05:28.580982 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.580909 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv" event={"ID":"722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b","Type":"ContainerDied","Data":"6ecff86a615cf52d5836870a5069680d53c42a92c27413a2af034ebd255f5247"} Oct 30 18:05:28.580982 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.580944 4965 scope.go:122] "RemoveContainer" containerID="55919ca08474befb4c087b7bd0d75e85118eb591dd5755982eb42a27ea686318" Oct 30 18:05:28.596303 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.596253 4965 scope.go:122] "RemoveContainer" containerID="55919ca08474befb4c087b7bd0d75e85118eb591dd5755982eb42a27ea686318" Oct 30 18:05:28.596721 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:05:28.596698 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55919ca08474befb4c087b7bd0d75e85118eb591dd5755982eb42a27ea686318\": container with ID starting with 55919ca08474befb4c087b7bd0d75e85118eb591dd5755982eb42a27ea686318 not found: ID does not exist" containerID="55919ca08474befb4c087b7bd0d75e85118eb591dd5755982eb42a27ea686318" Oct 30 18:05:28.596788 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.596732 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55919ca08474befb4c087b7bd0d75e85118eb591dd5755982eb42a27ea686318"} err="failed to get container status \"55919ca08474befb4c087b7bd0d75e85118eb591dd5755982eb42a27ea686318\": rpc error: code = NotFound desc = could not find container \"55919ca08474befb4c087b7bd0d75e85118eb591dd5755982eb42a27ea686318\": container with ID starting with 55919ca08474befb4c087b7bd0d75e85118eb591dd5755982eb42a27ea686318 not found: ID does not exist" Oct 30 18:05:28.652360 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.652327 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv"] Oct 30 18:05:28.663417 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.663389 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-2zgwv"] Oct 30 18:05:28.745043 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.739808 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6"] Oct 30 18:05:28.745043 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:28.740170 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" podUID="46a78684-48db-4c15-9122-0591ec698f82" containerName="istio-proxy" containerID="cri-o://cedd568980dd911b8cae6aace8ecfbc513a1f7fc8aa27afd44edb23efe5185bf" gracePeriod=30 Oct 30 18:05:30.099776 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:30.099741 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b" path="/var/lib/kubelet/pods/722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b/volumes" Oct 30 18:05:31.163761 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.163727 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2"] Oct 30 18:05:31.164283 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.163917 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2" podUID="d9c048dc-92d6-46c4-9ada-47b9b5d57e35" containerName="maas-api" containerID="cri-o://07ed131f183e55c3aae4aa5ffadbe84a858ba90b52507f2686b1b0e52d47859a" gracePeriod=30 Oct 30 18:05:31.372941 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.372907 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f10dda-658f6df5fb-n9xc5"] Oct 30 18:05:31.373167 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.373145 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-658f6df5fb-n9xc5" podUID="b75b13e4-a0a9-45a4-84f4-87082c3d0548" containerName="payload-processing" containerID="cri-o://7576743247ef5fc41fdcdf395b26877644743958a071da61f67bddef3d2c7007" gracePeriod=30 Oct 30 18:05:31.510355 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.510321 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2" Oct 30 18:05:31.610501 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.610464 4965 generic.go:356] "Generic (PLEG): container finished" podID="b75b13e4-a0a9-45a4-84f4-87082c3d0548" containerID="7576743247ef5fc41fdcdf395b26877644743958a071da61f67bddef3d2c7007" exitCode=0 Oct 30 18:05:31.610802 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.610548 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-658f6df5fb-n9xc5" event={"ID":"b75b13e4-a0a9-45a4-84f4-87082c3d0548","Type":"ContainerDied","Data":"7576743247ef5fc41fdcdf395b26877644743958a071da61f67bddef3d2c7007"} Oct 30 18:05:31.612989 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.612951 4965 generic.go:356] "Generic (PLEG): container finished" podID="d9c048dc-92d6-46c4-9ada-47b9b5d57e35" containerID="07ed131f183e55c3aae4aa5ffadbe84a858ba90b52507f2686b1b0e52d47859a" exitCode=0 Oct 30 18:05:31.613152 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.613033 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2" Oct 30 18:05:31.613243 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.613175 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2" event={"ID":"d9c048dc-92d6-46c4-9ada-47b9b5d57e35","Type":"ContainerDied","Data":"07ed131f183e55c3aae4aa5ffadbe84a858ba90b52507f2686b1b0e52d47859a"} Oct 30 18:05:31.613243 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.613223 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2" event={"ID":"d9c048dc-92d6-46c4-9ada-47b9b5d57e35","Type":"ContainerDied","Data":"fc0d8a375c1f7f98d6314a8ec8cefe8755ea9bb92d004f294b166f4c648af109"} Oct 30 18:05:31.613400 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.613254 4965 scope.go:122] "RemoveContainer" containerID="07ed131f183e55c3aae4aa5ffadbe84a858ba90b52507f2686b1b0e52d47859a" Oct 30 18:05:31.619413 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.619205 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9c048dc-92d6-46c4-9ada-47b9b5d57e35-kube-api-access-xbkl8\" (UniqueName: \"kubernetes.io/projected/d9c048dc-92d6-46c4-9ada-47b9b5d57e35-kube-api-access-xbkl8\") pod \"d9c048dc-92d6-46c4-9ada-47b9b5d57e35\" (UID: \"d9c048dc-92d6-46c4-9ada-47b9b5d57e35\") " Oct 30 18:05:31.619413 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.619251 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9c048dc-92d6-46c4-9ada-47b9b5d57e35-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c048dc-92d6-46c4-9ada-47b9b5d57e35-openshift-service-ca\") pod \"d9c048dc-92d6-46c4-9ada-47b9b5d57e35\" (UID: \"d9c048dc-92d6-46c4-9ada-47b9b5d57e35\") " Oct 30 18:05:31.619413 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.619317 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9c048dc-92d6-46c4-9ada-47b9b5d57e35-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9c048dc-92d6-46c4-9ada-47b9b5d57e35-maas-api-tls\") pod \"d9c048dc-92d6-46c4-9ada-47b9b5d57e35\" (UID: \"d9c048dc-92d6-46c4-9ada-47b9b5d57e35\") " Oct 30 18:05:31.620175 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.619615 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c048dc-92d6-46c4-9ada-47b9b5d57e35-openshift-service-ca" pod "d9c048dc-92d6-46c4-9ada-47b9b5d57e35" (UID: "d9c048dc-92d6-46c4-9ada-47b9b5d57e35"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:05:31.620175 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.619807 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9c048dc-92d6-46c4-9ada-47b9b5d57e35-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9c048dc-92d6-46c4-9ada-47b9b5d57e35-metrics-certs\") pod \"d9c048dc-92d6-46c4-9ada-47b9b5d57e35\" (UID: \"d9c048dc-92d6-46c4-9ada-47b9b5d57e35\") " Oct 30 18:05:31.620660 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.620639 4965 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c048dc-92d6-46c4-9ada-47b9b5d57e35-openshift-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:31.622479 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.622454 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9c048dc-92d6-46c4-9ada-47b9b5d57e35-kube-api-access-xbkl8" pod "d9c048dc-92d6-46c4-9ada-47b9b5d57e35" (UID: "d9c048dc-92d6-46c4-9ada-47b9b5d57e35"). InnerVolumeSpecName "kube-api-access-xbkl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:31.623478 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.623456 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9c048dc-92d6-46c4-9ada-47b9b5d57e35-metrics-certs" pod "d9c048dc-92d6-46c4-9ada-47b9b5d57e35" (UID: "d9c048dc-92d6-46c4-9ada-47b9b5d57e35"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:05:31.625191 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.625172 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9c048dc-92d6-46c4-9ada-47b9b5d57e35-maas-api-tls" pod "d9c048dc-92d6-46c4-9ada-47b9b5d57e35" (UID: "d9c048dc-92d6-46c4-9ada-47b9b5d57e35"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:05:31.628325 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.628294 4965 scope.go:122] "RemoveContainer" containerID="07ed131f183e55c3aae4aa5ffadbe84a858ba90b52507f2686b1b0e52d47859a" Oct 30 18:05:31.628638 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:05:31.628603 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07ed131f183e55c3aae4aa5ffadbe84a858ba90b52507f2686b1b0e52d47859a\": container with ID starting with 07ed131f183e55c3aae4aa5ffadbe84a858ba90b52507f2686b1b0e52d47859a not found: ID does not exist" containerID="07ed131f183e55c3aae4aa5ffadbe84a858ba90b52507f2686b1b0e52d47859a" Oct 30 18:05:31.628714 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.628642 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07ed131f183e55c3aae4aa5ffadbe84a858ba90b52507f2686b1b0e52d47859a"} err="failed to get container status \"07ed131f183e55c3aae4aa5ffadbe84a858ba90b52507f2686b1b0e52d47859a\": rpc error: code = NotFound desc = could not find container \"07ed131f183e55c3aae4aa5ffadbe84a858ba90b52507f2686b1b0e52d47859a\": container with ID starting with 07ed131f183e55c3aae4aa5ffadbe84a858ba90b52507f2686b1b0e52d47859a not found: ID does not exist" Oct 30 18:05:31.662145 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.662128 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-658f6df5fb-n9xc5" Oct 30 18:05:31.729941 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.729451 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b75b13e4-a0a9-45a4-84f4-87082c3d0548-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b75b13e4-a0a9-45a4-84f4-87082c3d0548-plugins-config-volume\") pod \"b75b13e4-a0a9-45a4-84f4-87082c3d0548\" (UID: \"b75b13e4-a0a9-45a4-84f4-87082c3d0548\") " Oct 30 18:05:31.729941 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.729577 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b75b13e4-a0a9-45a4-84f4-87082c3d0548-kube-api-access-j67fw\" (UniqueName: \"kubernetes.io/projected/b75b13e4-a0a9-45a4-84f4-87082c3d0548-kube-api-access-j67fw\") pod \"b75b13e4-a0a9-45a4-84f4-87082c3d0548\" (UID: \"b75b13e4-a0a9-45a4-84f4-87082c3d0548\") " Oct 30 18:05:31.730216 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.730150 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbkl8\" (UniqueName: \"kubernetes.io/projected/d9c048dc-92d6-46c4-9ada-47b9b5d57e35-kube-api-access-xbkl8\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:31.730216 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.730174 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9c048dc-92d6-46c4-9ada-47b9b5d57e35-maas-api-tls\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:31.730216 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.730189 4965 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9c048dc-92d6-46c4-9ada-47b9b5d57e35-metrics-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:31.742965 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.741431 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b75b13e4-a0a9-45a4-84f4-87082c3d0548-kube-api-access-j67fw" pod "b75b13e4-a0a9-45a4-84f4-87082c3d0548" (UID: "b75b13e4-a0a9-45a4-84f4-87082c3d0548"). InnerVolumeSpecName "kube-api-access-j67fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:31.743400 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.743371 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b75b13e4-a0a9-45a4-84f4-87082c3d0548-plugins-config-volume" pod "b75b13e4-a0a9-45a4-84f4-87082c3d0548" (UID: "b75b13e4-a0a9-45a4-84f4-87082c3d0548"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:05:31.832073 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.832038 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b75b13e4-a0a9-45a4-84f4-87082c3d0548-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:31.832073 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.832068 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j67fw\" (UniqueName: \"kubernetes.io/projected/b75b13e4-a0a9-45a4-84f4-87082c3d0548-kube-api-access-j67fw\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:31.963310 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.963275 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2"] Oct 30 18:05:31.967715 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:31.967685 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10dda-745fcd74d8-rrmk2"] Oct 30 18:05:32.115681 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:32.115645 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9c048dc-92d6-46c4-9ada-47b9b5d57e35" path="/var/lib/kubelet/pods/d9c048dc-92d6-46c4-9ada-47b9b5d57e35/volumes" Oct 30 18:05:32.622470 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:32.622435 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-658f6df5fb-n9xc5" Oct 30 18:05:32.622926 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:32.622440 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10dda-658f6df5fb-n9xc5" event={"ID":"b75b13e4-a0a9-45a4-84f4-87082c3d0548","Type":"ContainerDied","Data":"39689d8deb14fc77fe4c116226198de58fc9c8485778b140b20b41dfc061d81d"} Oct 30 18:05:32.622926 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:32.622571 4965 scope.go:122] "RemoveContainer" containerID="7576743247ef5fc41fdcdf395b26877644743958a071da61f67bddef3d2c7007" Oct 30 18:05:32.675474 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:32.675443 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f10dda-658f6df5fb-n9xc5"] Oct 30 18:05:32.687719 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:32.687691 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f10dda-658f6df5fb-n9xc5"] Oct 30 18:05:34.085481 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.085454 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 18:05:34.099079 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.099039 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b75b13e4-a0a9-45a4-84f4-87082c3d0548" path="/var/lib/kubelet/pods/b75b13e4-a0a9-45a4-84f4-87082c3d0548/volumes" Oct 30 18:05:34.165293 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.165257 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-workload-certs\") pod \"46a78684-48db-4c15-9122-0591ec698f82\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " Oct 30 18:05:34.165466 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.165370 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-istio-data\") pod \"46a78684-48db-4c15-9122-0591ec698f82\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " Oct 30 18:05:34.165466 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.165452 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46a78684-48db-4c15-9122-0591ec698f82-istio-token\" (UniqueName: \"kubernetes.io/projected/46a78684-48db-4c15-9122-0591ec698f82-istio-token\") pod \"46a78684-48db-4c15-9122-0591ec698f82\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " Oct 30 18:05:34.165613 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.165590 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46a78684-48db-4c15-9122-0591ec698f82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46a78684-48db-4c15-9122-0591ec698f82-istiod-ca-cert\") pod \"46a78684-48db-4c15-9122-0591ec698f82\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " Oct 30 18:05:34.165771 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.165747 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-istio-envoy\") pod \"46a78684-48db-4c15-9122-0591ec698f82\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " Oct 30 18:05:34.165842 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.165785 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-credential-socket\") pod \"46a78684-48db-4c15-9122-0591ec698f82\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " Oct 30 18:05:34.165842 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.165634 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-workload-certs" pod "46a78684-48db-4c15-9122-0591ec698f82" (UID: "46a78684-48db-4c15-9122-0591ec698f82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:34.165842 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.165823 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46a78684-48db-4c15-9122-0591ec698f82-kube-api-access-87z6x\" (UniqueName: \"kubernetes.io/projected/46a78684-48db-4c15-9122-0591ec698f82-kube-api-access-87z6x\") pod \"46a78684-48db-4c15-9122-0591ec698f82\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " Oct 30 18:05:34.166037 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.165896 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-workload-socket\") pod \"46a78684-48db-4c15-9122-0591ec698f82\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " Oct 30 18:05:34.166037 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.165959 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46a78684-48db-4c15-9122-0591ec698f82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46a78684-48db-4c15-9122-0591ec698f82-istio-podinfo\") pod \"46a78684-48db-4c15-9122-0591ec698f82\" (UID: \"46a78684-48db-4c15-9122-0591ec698f82\") " Oct 30 18:05:34.166160 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.165799 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-istio-data" pod "46a78684-48db-4c15-9122-0591ec698f82" (UID: "46a78684-48db-4c15-9122-0591ec698f82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:34.166217 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.166067 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46a78684-48db-4c15-9122-0591ec698f82-istiod-ca-cert" pod "46a78684-48db-4c15-9122-0591ec698f82" (UID: "46a78684-48db-4c15-9122-0591ec698f82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:05:34.166403 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.166377 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-credential-socket" pod "46a78684-48db-4c15-9122-0591ec698f82" (UID: "46a78684-48db-4c15-9122-0591ec698f82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:34.166708 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.166689 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:34.166708 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.166709 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:34.166942 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.166721 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:34.166942 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.166733 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46a78684-48db-4c15-9122-0591ec698f82-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:34.166942 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.166825 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-workload-socket" pod "46a78684-48db-4c15-9122-0591ec698f82" (UID: "46a78684-48db-4c15-9122-0591ec698f82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:34.169063 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.169034 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46a78684-48db-4c15-9122-0591ec698f82-kube-api-access-87z6x" pod "46a78684-48db-4c15-9122-0591ec698f82" (UID: "46a78684-48db-4c15-9122-0591ec698f82"). InnerVolumeSpecName "kube-api-access-87z6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:34.169192 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.169120 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46a78684-48db-4c15-9122-0591ec698f82-istio-token" pod "46a78684-48db-4c15-9122-0591ec698f82" (UID: "46a78684-48db-4c15-9122-0591ec698f82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:34.169299 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.169283 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46a78684-48db-4c15-9122-0591ec698f82-istio-podinfo" pod "46a78684-48db-4c15-9122-0591ec698f82" (UID: "46a78684-48db-4c15-9122-0591ec698f82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:05:34.170074 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.170051 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-istio-envoy" pod "46a78684-48db-4c15-9122-0591ec698f82" (UID: "46a78684-48db-4c15-9122-0591ec698f82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:34.178985 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.178961 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp"] Oct 30 18:05:34.196591 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.196563 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj"] Oct 30 18:05:34.268172 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.268144 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46a78684-48db-4c15-9122-0591ec698f82-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:34.268172 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.268170 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:34.268369 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.268181 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87z6x\" (UniqueName: \"kubernetes.io/projected/46a78684-48db-4c15-9122-0591ec698f82-kube-api-access-87z6x\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:34.268369 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.268191 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a78684-48db-4c15-9122-0591ec698f82-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:34.268369 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.268201 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46a78684-48db-4c15-9122-0591ec698f82-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:34.640288 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.640257 4965 generic.go:356] "Generic (PLEG): container finished" podID="46a78684-48db-4c15-9122-0591ec698f82" containerID="cedd568980dd911b8cae6aace8ecfbc513a1f7fc8aa27afd44edb23efe5185bf" exitCode=0 Oct 30 18:05:34.640492 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.640305 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" Oct 30 18:05:34.640492 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.640337 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" event={"ID":"46a78684-48db-4c15-9122-0591ec698f82","Type":"ContainerDied","Data":"cedd568980dd911b8cae6aace8ecfbc513a1f7fc8aa27afd44edb23efe5185bf"} Oct 30 18:05:34.640492 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.640392 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6" event={"ID":"46a78684-48db-4c15-9122-0591ec698f82","Type":"ContainerDied","Data":"c5d117d27498b5ac84ebe56a0550cdabbbfbb1e62a6141abf572f549129a3ffa"} Oct 30 18:05:34.640492 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.640419 4965 scope.go:122] "RemoveContainer" containerID="cedd568980dd911b8cae6aace8ecfbc513a1f7fc8aa27afd44edb23efe5185bf" Oct 30 18:05:34.652710 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.652691 4965 scope.go:122] "RemoveContainer" containerID="cedd568980dd911b8cae6aace8ecfbc513a1f7fc8aa27afd44edb23efe5185bf" Oct 30 18:05:34.653062 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:05:34.653038 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cedd568980dd911b8cae6aace8ecfbc513a1f7fc8aa27afd44edb23efe5185bf\": container with ID starting with cedd568980dd911b8cae6aace8ecfbc513a1f7fc8aa27afd44edb23efe5185bf not found: ID does not exist" containerID="cedd568980dd911b8cae6aace8ecfbc513a1f7fc8aa27afd44edb23efe5185bf" Oct 30 18:05:34.653153 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.653069 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cedd568980dd911b8cae6aace8ecfbc513a1f7fc8aa27afd44edb23efe5185bf"} err="failed to get container status \"cedd568980dd911b8cae6aace8ecfbc513a1f7fc8aa27afd44edb23efe5185bf\": rpc error: code = NotFound desc = could not find container \"cedd568980dd911b8cae6aace8ecfbc513a1f7fc8aa27afd44edb23efe5185bf\": container with ID starting with cedd568980dd911b8cae6aace8ecfbc513a1f7fc8aa27afd44edb23efe5185bf not found: ID does not exist" Oct 30 18:05:34.704373 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.704341 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6"] Oct 30 18:05:34.724174 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:34.724139 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-01b925-openshift-default-5f48b54fbf-8vqs6"] Oct 30 18:05:35.999755 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:05:35.999706 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:05:36.099979 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:36.099946 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46a78684-48db-4c15-9122-0591ec698f82" path="/var/lib/kubelet/pods/46a78684-48db-4c15-9122-0591ec698f82/volumes" Oct 30 18:05:39.367331 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.367300 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-6dcd6678f9-jjjkd"] Oct 30 18:05:39.369469 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.369440 4965 cpu_manager.go:414] "containerMap: removing container" podUID="404cc4e8-d6c9-4d5b-8abe-c68527da5248" containerName="payload-pre-processing" Oct 30 18:05:39.369649 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.369635 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="404cc4e8-d6c9-4d5b-8abe-c68527da5248" containerName="payload-pre-processing" Oct 30 18:05:39.369756 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.369744 4965 cpu_manager.go:414] "containerMap: removing container" podUID="78b12271-78e7-4b9f-98be-cc3d6b736708" containerName="payload-processing" Oct 30 18:05:39.369994 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.369825 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78b12271-78e7-4b9f-98be-cc3d6b736708" containerName="payload-processing" Oct 30 18:05:39.370117 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.370107 4965 cpu_manager.go:414] "containerMap: removing container" podUID="d9c048dc-92d6-46c4-9ada-47b9b5d57e35" containerName="maas-api" Oct 30 18:05:39.370190 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.370181 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9c048dc-92d6-46c4-9ada-47b9b5d57e35" containerName="maas-api" Oct 30 18:05:39.370278 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.370268 4965 cpu_manager.go:414] "containerMap: removing container" podUID="d022f31a-8a4f-4a1c-9b99-2b53227e9d49" containerName="payload-pre-processing" Oct 30 18:05:39.370351 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.370342 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d022f31a-8a4f-4a1c-9b99-2b53227e9d49" containerName="payload-pre-processing" Oct 30 18:05:39.370431 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.370422 4965 cpu_manager.go:414] "containerMap: removing container" podUID="40110c67-890b-40ea-ad4c-0c673666f1cd" containerName="maas-api" Oct 30 18:05:39.370498 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.370488 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40110c67-890b-40ea-ad4c-0c673666f1cd" containerName="maas-api" Oct 30 18:05:39.370583 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.370574 4965 cpu_manager.go:414] "containerMap: removing container" podUID="8abc8140-8a36-4cc7-80da-43738e049fe0" containerName="payload-processing" Oct 30 18:05:39.370649 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.370641 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8abc8140-8a36-4cc7-80da-43738e049fe0" containerName="payload-processing" Oct 30 18:05:39.370728 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.370719 4965 cpu_manager.go:414] "containerMap: removing container" podUID="dcb3f0ba-021a-4788-ad64-fcc9054fe5a1" containerName="payload-pre-processing" Oct 30 18:05:39.370798 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.370789 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcb3f0ba-021a-4788-ad64-fcc9054fe5a1" containerName="payload-pre-processing" Oct 30 18:05:39.371013 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.370968 4965 cpu_manager.go:414] "containerMap: removing container" podUID="b48e4e05-e95d-424c-887e-668bdc42e398" containerName="istio-proxy" Oct 30 18:05:39.371156 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.371139 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b48e4e05-e95d-424c-887e-668bdc42e398" containerName="istio-proxy" Oct 30 18:05:39.371325 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.371251 4965 cpu_manager.go:414] "containerMap: removing container" podUID="722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b" containerName="istio-proxy" Oct 30 18:05:39.371428 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.371413 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b" containerName="istio-proxy" Oct 30 18:05:39.371527 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.371516 4965 cpu_manager.go:414] "containerMap: removing container" podUID="b75b13e4-a0a9-45a4-84f4-87082c3d0548" containerName="payload-processing" Oct 30 18:05:39.371604 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.371594 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b75b13e4-a0a9-45a4-84f4-87082c3d0548" containerName="payload-processing" Oct 30 18:05:39.371689 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.371681 4965 cpu_manager.go:414] "containerMap: removing container" podUID="2edd6ff6-320a-42d4-83ef-5173fcbf7b92" containerName="istio-proxy" Oct 30 18:05:39.371761 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.371751 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2edd6ff6-320a-42d4-83ef-5173fcbf7b92" containerName="istio-proxy" Oct 30 18:05:39.371850 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.371841 4965 cpu_manager.go:414] "containerMap: removing container" podUID="46a78684-48db-4c15-9122-0591ec698f82" containerName="istio-proxy" Oct 30 18:05:39.371942 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.371932 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46a78684-48db-4c15-9122-0591ec698f82" containerName="istio-proxy" Oct 30 18:05:39.372028 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.372019 4965 cpu_manager.go:414] "containerMap: removing container" podUID="cb5ef536-ab9a-423e-bea3-dac81549b81c" containerName="maas-api" Oct 30 18:05:39.372096 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.372087 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb5ef536-ab9a-423e-bea3-dac81549b81c" containerName="maas-api" Oct 30 18:05:39.372270 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.372259 4965 cpu_manager.go:414] "containerMap: removing container" podUID="24f2394c-1e9c-417e-91c0-5086ebae37c0" containerName="payload-pre-processing" Oct 30 18:05:39.372382 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.372371 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24f2394c-1e9c-417e-91c0-5086ebae37c0" containerName="payload-pre-processing" Oct 30 18:05:39.372467 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.372459 4965 cpu_manager.go:414] "containerMap: removing container" podUID="d174a74f-7205-4020-94b0-e30652c9b721" containerName="main" Oct 30 18:05:39.372542 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.372533 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d174a74f-7205-4020-94b0-e30652c9b721" containerName="main" Oct 30 18:05:39.372618 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.372610 4965 cpu_manager.go:414] "containerMap: removing container" podUID="4bd72d29-e1f9-49cd-8388-156d3796563a" containerName="payload-processing" Oct 30 18:05:39.372688 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.372679 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bd72d29-e1f9-49cd-8388-156d3796563a" containerName="payload-processing" Oct 30 18:05:39.372763 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.372755 4965 cpu_manager.go:414] "containerMap: removing container" podUID="d43a70eb-9850-4990-8385-fd2e2aab5bc2" containerName="istio-proxy" Oct 30 18:05:39.372829 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.372820 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d43a70eb-9850-4990-8385-fd2e2aab5bc2" containerName="istio-proxy" Oct 30 18:05:39.372938 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.372927 4965 cpu_manager.go:414] "containerMap: removing container" podUID="241c1434-3e5c-4120-8eb5-507bf30a92ec" containerName="istio-proxy" Oct 30 18:05:39.373034 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.373021 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="241c1434-3e5c-4120-8eb5-507bf30a92ec" containerName="istio-proxy" Oct 30 18:05:39.373150 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.373133 4965 cpu_manager.go:414] "containerMap: removing container" podUID="f2721707-097c-4adf-a415-15be655b9729" containerName="payload-processing" Oct 30 18:05:39.373357 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.373341 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2721707-097c-4adf-a415-15be655b9729" containerName="payload-processing" Oct 30 18:05:39.373470 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.373459 4965 cpu_manager.go:414] "containerMap: removing container" podUID="b48e4e05-e95d-424c-887e-668bdc42e398" containerName="istio-proxy" Oct 30 18:05:39.373551 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.373540 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b48e4e05-e95d-424c-887e-668bdc42e398" containerName="istio-proxy" Oct 30 18:05:39.373830 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.373816 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="404cc4e8-d6c9-4d5b-8abe-c68527da5248" containerName="payload-pre-processing" Oct 30 18:05:39.373969 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.373957 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="78b12271-78e7-4b9f-98be-cc3d6b736708" containerName="payload-processing" Oct 30 18:05:39.374060 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.374052 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9c048dc-92d6-46c4-9ada-47b9b5d57e35" containerName="maas-api" Oct 30 18:05:39.374144 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.374136 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="d022f31a-8a4f-4a1c-9b99-2b53227e9d49" containerName="payload-pre-processing" Oct 30 18:05:39.374247 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.374227 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="40110c67-890b-40ea-ad4c-0c673666f1cd" containerName="maas-api" Oct 30 18:05:39.374337 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.374328 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="8abc8140-8a36-4cc7-80da-43738e049fe0" containerName="payload-processing" Oct 30 18:05:39.374403 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.374394 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcb3f0ba-021a-4788-ad64-fcc9054fe5a1" containerName="payload-pre-processing" Oct 30 18:05:39.374478 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.374468 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="b48e4e05-e95d-424c-887e-668bdc42e398" containerName="istio-proxy" Oct 30 18:05:39.374557 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.374541 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="722e3954-7fe4-47c2-90f5-5d3b4eb5ca5b" containerName="istio-proxy" Oct 30 18:05:39.374557 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.374557 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="b75b13e4-a0a9-45a4-84f4-87082c3d0548" containerName="payload-processing" Oct 30 18:05:39.374701 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.374568 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="2edd6ff6-320a-42d4-83ef-5173fcbf7b92" containerName="istio-proxy" Oct 30 18:05:39.374701 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.374579 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="46a78684-48db-4c15-9122-0591ec698f82" containerName="istio-proxy" Oct 30 18:05:39.374701 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.374590 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb5ef536-ab9a-423e-bea3-dac81549b81c" containerName="maas-api" Oct 30 18:05:39.374701 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.374598 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="24f2394c-1e9c-417e-91c0-5086ebae37c0" containerName="payload-pre-processing" Oct 30 18:05:39.374701 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.374611 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="d174a74f-7205-4020-94b0-e30652c9b721" containerName="main" Oct 30 18:05:39.374701 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.374621 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="2edd6ff6-320a-42d4-83ef-5173fcbf7b92" containerName="istio-proxy" Oct 30 18:05:39.374701 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.374628 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bd72d29-e1f9-49cd-8388-156d3796563a" containerName="payload-processing" Oct 30 18:05:39.374701 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.374638 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="d43a70eb-9850-4990-8385-fd2e2aab5bc2" containerName="istio-proxy" Oct 30 18:05:39.374701 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.374649 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="241c1434-3e5c-4120-8eb5-507bf30a92ec" containerName="istio-proxy" Oct 30 18:05:39.374701 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.374659 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2721707-097c-4adf-a415-15be655b9729" containerName="payload-processing" Oct 30 18:05:39.374701 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.374672 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="b48e4e05-e95d-424c-887e-668bdc42e398" containerName="istio-proxy" Oct 30 18:05:39.381572 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.380278 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-6dcd6678f9-jjjkd" Oct 30 18:05:39.419904 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.396696 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-96c514\"" Oct 30 18:05:39.419904 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.397428 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-96c514-dockercfg-tqn4l\"" Oct 30 18:05:39.424943 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.424914 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-6dcd6678f9-jjjkd"] Oct 30 18:05:39.473992 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.473962 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96c514-5cd48b54dd-8tn7q"] Oct 30 18:05:39.474660 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.474635 4965 cpu_manager.go:414] "containerMap: removing container" podUID="2edd6ff6-320a-42d4-83ef-5173fcbf7b92" containerName="istio-proxy" Oct 30 18:05:39.474660 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.474651 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2edd6ff6-320a-42d4-83ef-5173fcbf7b92" containerName="istio-proxy" Oct 30 18:05:39.478318 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.478299 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-5cd48b54dd-8tn7q" Oct 30 18:05:39.498085 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.496845 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96c514-5cd48b54dd-8tn7q"] Oct 30 18:05:39.498085 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.497762 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh8kr\" (UniqueName: \"kubernetes.io/projected/2c52d606-3e47-4e08-bb5d-4f018e638134-kube-api-access-wh8kr\") pod \"payload-pre-processing-e2e-ipp-b-96c514-6dcd6678f9-jjjkd\" (UID: \"2c52d606-3e47-4e08-bb5d-4f018e638134\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-6dcd6678f9-jjjkd" Oct 30 18:05:39.498085 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.497832 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c52d606-3e47-4e08-bb5d-4f018e638134-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-96c514-6dcd6678f9-jjjkd\" (UID: \"2c52d606-3e47-4e08-bb5d-4f018e638134\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-6dcd6678f9-jjjkd" Oct 30 18:05:39.599476 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.599447 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42a77847-5a9e-4825-80db-5f0b19133f62-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-96c514-5cd48b54dd-8tn7q\" (UID: \"42a77847-5a9e-4825-80db-5f0b19133f62\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-5cd48b54dd-8tn7q" Oct 30 18:05:39.599628 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.599525 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh8kr\" (UniqueName: \"kubernetes.io/projected/2c52d606-3e47-4e08-bb5d-4f018e638134-kube-api-access-wh8kr\") pod \"payload-pre-processing-e2e-ipp-b-96c514-6dcd6678f9-jjjkd\" (UID: \"2c52d606-3e47-4e08-bb5d-4f018e638134\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-6dcd6678f9-jjjkd" Oct 30 18:05:39.599628 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.599594 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c52d606-3e47-4e08-bb5d-4f018e638134-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-96c514-6dcd6678f9-jjjkd\" (UID: \"2c52d606-3e47-4e08-bb5d-4f018e638134\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-6dcd6678f9-jjjkd" Oct 30 18:05:39.599729 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.599652 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8m85\" (UniqueName: \"kubernetes.io/projected/42a77847-5a9e-4825-80db-5f0b19133f62-kube-api-access-k8m85\") pod \"payload-processing-e2e-ipp-b-96c514-5cd48b54dd-8tn7q\" (UID: \"42a77847-5a9e-4825-80db-5f0b19133f62\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-5cd48b54dd-8tn7q" Oct 30 18:05:39.600364 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.600343 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c52d606-3e47-4e08-bb5d-4f018e638134-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-96c514-6dcd6678f9-jjjkd\" (UID: \"2c52d606-3e47-4e08-bb5d-4f018e638134\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-6dcd6678f9-jjjkd" Oct 30 18:05:39.622387 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.622329 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh8kr\" (UniqueName: \"kubernetes.io/projected/2c52d606-3e47-4e08-bb5d-4f018e638134-kube-api-access-wh8kr\") pod \"payload-pre-processing-e2e-ipp-b-96c514-6dcd6678f9-jjjkd\" (UID: \"2c52d606-3e47-4e08-bb5d-4f018e638134\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-6dcd6678f9-jjjkd" Oct 30 18:05:39.701200 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.701172 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8m85\" (UniqueName: \"kubernetes.io/projected/42a77847-5a9e-4825-80db-5f0b19133f62-kube-api-access-k8m85\") pod \"payload-processing-e2e-ipp-b-96c514-5cd48b54dd-8tn7q\" (UID: \"42a77847-5a9e-4825-80db-5f0b19133f62\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-5cd48b54dd-8tn7q" Oct 30 18:05:39.701353 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.701293 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42a77847-5a9e-4825-80db-5f0b19133f62-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-96c514-5cd48b54dd-8tn7q\" (UID: \"42a77847-5a9e-4825-80db-5f0b19133f62\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-5cd48b54dd-8tn7q" Oct 30 18:05:39.701944 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.701928 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42a77847-5a9e-4825-80db-5f0b19133f62-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-96c514-5cd48b54dd-8tn7q\" (UID: \"42a77847-5a9e-4825-80db-5f0b19133f62\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-5cd48b54dd-8tn7q" Oct 30 18:05:39.724099 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.724075 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8m85\" (UniqueName: \"kubernetes.io/projected/42a77847-5a9e-4825-80db-5f0b19133f62-kube-api-access-k8m85\") pod \"payload-processing-e2e-ipp-b-96c514-5cd48b54dd-8tn7q\" (UID: \"42a77847-5a9e-4825-80db-5f0b19133f62\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-5cd48b54dd-8tn7q" Oct 30 18:05:39.729984 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.729963 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-6dcd6678f9-jjjkd" Oct 30 18:05:39.793170 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.793146 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-5cd48b54dd-8tn7q" Oct 30 18:05:39.943733 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:39.943696 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-455086-6c947b47-fllqz"] Oct 30 18:05:40.074993 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:40.074444 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-6dcd6678f9-jjjkd"] Oct 30 18:05:40.126174 ip-10-0-139-222 kubenswrapper[4965]: W1030 18:05:40.126148 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a77847_5a9e_4825_80db_5f0b19133f62.slice/crio-41f221d93adaaab55a6a2d656b41594243be7b507643fb4dfbbf8d2a949117a1 WatchSource:0}: Error finding container 41f221d93adaaab55a6a2d656b41594243be7b507643fb4dfbbf8d2a949117a1: Status 404 returned error can't find the container with id 41f221d93adaaab55a6a2d656b41594243be7b507643fb4dfbbf8d2a949117a1 Oct 30 18:05:40.126608 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:40.126559 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96c514-5cd48b54dd-8tn7q"] Oct 30 18:05:40.693063 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:40.693025 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-6dcd6678f9-jjjkd" event={"ID":"2c52d606-3e47-4e08-bb5d-4f018e638134","Type":"ContainerStarted","Data":"de5f4896d5fcd8288631835d0a90ab7d2375f0f6b05df8f5aaa00b547f323a38"} Oct 30 18:05:40.697507 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:40.697460 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-6dcd6678f9-jjjkd" event={"ID":"2c52d606-3e47-4e08-bb5d-4f018e638134","Type":"ContainerStarted","Data":"4431e4829532fdaa772f7ce75670e5a4235c6deab1215a4cf4c837f2cb53f51e"} Oct 30 18:05:40.697998 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:40.697976 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-6dcd6678f9-jjjkd" Oct 30 18:05:40.700099 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:40.700014 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-5cd48b54dd-8tn7q" event={"ID":"42a77847-5a9e-4825-80db-5f0b19133f62","Type":"ContainerStarted","Data":"a10739ec5ad178afa0aa36a86727d7b064b18c5021d688377638ede7d034c1c1"} Oct 30 18:05:40.700099 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:40.700057 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-5cd48b54dd-8tn7q" event={"ID":"42a77847-5a9e-4825-80db-5f0b19133f62","Type":"ContainerStarted","Data":"41f221d93adaaab55a6a2d656b41594243be7b507643fb4dfbbf8d2a949117a1"} Oct 30 18:05:40.700748 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:40.700329 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-5cd48b54dd-8tn7q" Oct 30 18:05:40.789974 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:40.789906 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-6dcd6678f9-jjjkd" podStartSLOduration=1.789887869 podStartE2EDuration="1.789887869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:40.747910392 +0000 UTC m=+3198.800010323" watchObservedRunningTime="2026-10-30 18:05:40.789887869 +0000 UTC m=+3198.841987803" Oct 30 18:05:40.792670 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:40.792622 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-5cd48b54dd-8tn7q" podStartSLOduration=1.792605971 podStartE2EDuration="1.792605971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:40.788649626 +0000 UTC m=+3198.840749559" watchObservedRunningTime="2026-10-30 18:05:40.792605971 +0000 UTC m=+3198.844705903" Oct 30 18:05:43.671794 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:43.671759 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-d207f8-d8677dc5d-5pgjt"] Oct 30 18:05:44.944309 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:44.944273 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-455086-6c947b47-fllqz" podUID="dfcedd36-934a-4257-b1e9-db368ed2bcd5" containerName="payload-pre-processing" containerID="cri-o://1a95329f71d22d5c904d568dae3ca624bb25ed85e95cf68a0ed5eba499cadb72" gracePeriod=25 Oct 30 18:05:48.724103 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:48.724062 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-d207f8-d8677dc5d-5pgjt" podUID="6706cbd9-edfa-4b4d-ae4a-44a037e61efc" containerName="payload-processing" containerID="cri-o://213ec240dba617b0f3d51ad3b068504c54a851a62bf770f48a19e5c665cf5794" gracePeriod=25 Oct 30 18:05:49.201852 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.201783 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" containerID="cri-o://de3f8588bd9afe41c9bde81c2e70045c698e7c6102bc9070fb2f868b35df7887" gracePeriod=45 Oct 30 18:05:49.218313 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.218256 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" containerID="cri-o://51e78be71752c6b09881c3f34796c886b9a03d712a5ca0b5cde7e1d0d0a659c7" gracePeriod=45 Oct 30 18:05:49.222191 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.222161 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": dial tcp 10.132.0.65:8000: connect: connection refused" Oct 30 18:05:49.523114 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.523085 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 18:05:49.526473 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.526454 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 18:05:49.628255 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.628217 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-dshm\") pod \"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d\" (UID: \"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d\") " Oct 30 18:05:49.628447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.628302 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-tmp-dir\") pod \"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4\" (UID: \"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4\") " Oct 30 18:05:49.628447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.628335 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-kube-api-access-z8dq8\" (UniqueName: \"kubernetes.io/projected/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-kube-api-access-z8dq8\") pod \"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d\" (UID: \"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d\") " Oct 30 18:05:49.628447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.628390 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-home\" (UniqueName: \"kubernetes.io/empty-dir/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-home\") pod \"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d\" (UID: \"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d\") " Oct 30 18:05:49.628645 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.628459 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-tls-certs\" (UniqueName: \"kubernetes.io/secret/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-tls-certs\") pod \"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4\" (UID: \"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4\") " Oct 30 18:05:49.628645 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.628523 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-model-cache\") pod \"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4\" (UID: \"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4\") " Oct 30 18:05:49.628645 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.628597 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-home\" (UniqueName: \"kubernetes.io/empty-dir/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-home\") pod \"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4\" (UID: \"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4\") " Oct 30 18:05:49.628823 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.628723 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-kube-api-access-vdsvx\" (UniqueName: \"kubernetes.io/projected/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-kube-api-access-vdsvx\") pod \"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4\" (UID: \"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4\") " Oct 30 18:05:49.628823 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.628816 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-tmp-dir\") pod \"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d\" (UID: \"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d\") " Oct 30 18:05:49.628962 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.628857 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-dshm\") pod \"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4\" (UID: \"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4\") " Oct 30 18:05:49.628962 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.628921 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-tls-certs\" (UniqueName: \"kubernetes.io/secret/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-tls-certs\") pod \"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d\" (UID: \"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d\") " Oct 30 18:05:49.628962 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.628954 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-model-cache\") pod \"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d\" (UID: \"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d\") " Oct 30 18:05:49.629846 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.629815 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-model-cache" pod "38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" (UID: "38c90d17-2efa-4b7c-b10f-06a3a8b2a13d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:49.631234 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.630166 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-model-cache" pod "bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" (UID: "bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:49.631234 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.630172 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-tmp-dir" pod "38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" (UID: "38c90d17-2efa-4b7c-b10f-06a3a8b2a13d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:49.631234 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.630327 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-home" pod "38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" (UID: "38c90d17-2efa-4b7c-b10f-06a3a8b2a13d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:49.631234 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.630403 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-tmp-dir" pod "bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" (UID: "bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:49.631234 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.630420 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-home" pod "bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" (UID: "bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:49.633247 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.633218 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-dshm" pod "38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" (UID: "38c90d17-2efa-4b7c-b10f-06a3a8b2a13d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:49.633437 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.633406 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-dshm" pod "bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" (UID: "bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:49.637483 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.634010 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-tls-certs" pod "38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" (UID: "38c90d17-2efa-4b7c-b10f-06a3a8b2a13d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:05:49.637483 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.638309 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-kube-api-access-vdsvx" pod "bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" (UID: "bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4"). InnerVolumeSpecName "kube-api-access-vdsvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:49.637483 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.638469 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-tls-certs" pod "bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" (UID: "bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:05:49.637483 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.640032 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-kube-api-access-z8dq8" pod "38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" (UID: "38c90d17-2efa-4b7c-b10f-06a3a8b2a13d"). InnerVolumeSpecName "kube-api-access-z8dq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:49.730948 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.730915 4965 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-model-cache\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:49.730948 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.730944 4965 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-home\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:49.730948 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.730953 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdsvx\" (UniqueName: \"kubernetes.io/projected/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-kube-api-access-vdsvx\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:49.731404 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.730963 4965 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-tmp-dir\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:49.731404 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.730975 4965 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-dshm\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:49.731404 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.730982 4965 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-tls-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:49.731404 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.731001 4965 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-model-cache\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:49.731404 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.731009 4965 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-dshm\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:49.731404 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.731016 4965 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-tmp-dir\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:49.731404 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.731024 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8dq8\" (UniqueName: \"kubernetes.io/projected/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-kube-api-access-z8dq8\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:49.731404 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.731031 4965 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d-home\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:49.731404 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.731040 4965 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4-tls-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:49.768892 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.768782 4965 generic.go:356] "Generic (PLEG): container finished" podID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerID="de3f8588bd9afe41c9bde81c2e70045c698e7c6102bc9070fb2f868b35df7887" exitCode=0 Oct 30 18:05:49.768892 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.768825 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" Oct 30 18:05:49.768892 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.768877 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" event={"ID":"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d","Type":"ContainerDied","Data":"de3f8588bd9afe41c9bde81c2e70045c698e7c6102bc9070fb2f868b35df7887"} Oct 30 18:05:49.769150 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.768923 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" event={"ID":"38c90d17-2efa-4b7c-b10f-06a3a8b2a13d","Type":"ContainerDied","Data":"9dd73286f8c67b9bafa63961b2bbcaaeccbcf3c3745b4deb029faa4a304eac0e"} Oct 30 18:05:49.769150 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.768944 4965 scope.go:122] "RemoveContainer" containerID="de3f8588bd9afe41c9bde81c2e70045c698e7c6102bc9070fb2f868b35df7887" Oct 30 18:05:49.775884 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.775813 4965 generic.go:356] "Generic (PLEG): container finished" podID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerID="51e78be71752c6b09881c3f34796c886b9a03d712a5ca0b5cde7e1d0d0a659c7" exitCode=0 Oct 30 18:05:49.776360 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.776147 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" event={"ID":"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4","Type":"ContainerDied","Data":"51e78be71752c6b09881c3f34796c886b9a03d712a5ca0b5cde7e1d0d0a659c7"} Oct 30 18:05:49.776360 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.776329 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" event={"ID":"bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4","Type":"ContainerDied","Data":"e5a7923cb7f7a82602c342d86d1b98cb0409cb302d1dd09ff470aa81e4114f00"} Oct 30 18:05:49.776587 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.776251 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj" Oct 30 18:05:49.786786 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.786721 4965 scope.go:122] "RemoveContainer" containerID="de3f8588bd9afe41c9bde81c2e70045c698e7c6102bc9070fb2f868b35df7887" Oct 30 18:05:49.787345 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:05:49.787224 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de3f8588bd9afe41c9bde81c2e70045c698e7c6102bc9070fb2f868b35df7887\": container with ID starting with de3f8588bd9afe41c9bde81c2e70045c698e7c6102bc9070fb2f868b35df7887 not found: ID does not exist" containerID="de3f8588bd9afe41c9bde81c2e70045c698e7c6102bc9070fb2f868b35df7887" Oct 30 18:05:49.787345 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.787258 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de3f8588bd9afe41c9bde81c2e70045c698e7c6102bc9070fb2f868b35df7887"} err="failed to get container status \"de3f8588bd9afe41c9bde81c2e70045c698e7c6102bc9070fb2f868b35df7887\": rpc error: code = NotFound desc = could not find container \"de3f8588bd9afe41c9bde81c2e70045c698e7c6102bc9070fb2f868b35df7887\": container with ID starting with de3f8588bd9afe41c9bde81c2e70045c698e7c6102bc9070fb2f868b35df7887 not found: ID does not exist" Oct 30 18:05:49.787345 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.787283 4965 scope.go:122] "RemoveContainer" containerID="51e78be71752c6b09881c3f34796c886b9a03d712a5ca0b5cde7e1d0d0a659c7" Oct 30 18:05:49.798816 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.798790 4965 scope.go:122] "RemoveContainer" containerID="51e78be71752c6b09881c3f34796c886b9a03d712a5ca0b5cde7e1d0d0a659c7" Oct 30 18:05:49.799108 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:05:49.799087 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51e78be71752c6b09881c3f34796c886b9a03d712a5ca0b5cde7e1d0d0a659c7\": container with ID starting with 51e78be71752c6b09881c3f34796c886b9a03d712a5ca0b5cde7e1d0d0a659c7 not found: ID does not exist" containerID="51e78be71752c6b09881c3f34796c886b9a03d712a5ca0b5cde7e1d0d0a659c7" Oct 30 18:05:49.799195 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.799112 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51e78be71752c6b09881c3f34796c886b9a03d712a5ca0b5cde7e1d0d0a659c7"} err="failed to get container status \"51e78be71752c6b09881c3f34796c886b9a03d712a5ca0b5cde7e1d0d0a659c7\": rpc error: code = NotFound desc = could not find container \"51e78be71752c6b09881c3f34796c886b9a03d712a5ca0b5cde7e1d0d0a659c7\": container with ID starting with 51e78be71752c6b09881c3f34796c886b9a03d712a5ca0b5cde7e1d0d0a659c7 not found: ID does not exist" Oct 30 18:05:49.852404 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.852375 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp"] Oct 30 18:05:49.867334 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.867312 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp"] Oct 30 18:05:49.916614 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.916580 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj"] Oct 30 18:05:49.921293 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:49.921268 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-455086/premium-w1-455086-kserve-747bfccf85-wv5dj"] Oct 30 18:05:50.099834 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:50.099801 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" path="/var/lib/kubelet/pods/38c90d17-2efa-4b7c-b10f-06a3a8b2a13d/volumes" Oct 30 18:05:50.100322 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:50.100304 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" path="/var/lib/kubelet/pods/bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4/volumes" Oct 30 18:05:50.437136 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:50.437038 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-455086/distinct-2-w1-455086-kserve-5c574bf567-nv4kp" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 18:05:50.476585 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:50.476544 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k"] Oct 30 18:05:50.476953 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:50.476923 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" podUID="b003d430-64cd-4a08-9a96-6b326a534b5e" containerName="istio-proxy" containerID="cri-o://efb393c33e8a75e090b7e950d53b95bd27c15cf059e5ca90be6ad2119efd36f6" gracePeriod=30 Oct 30 18:05:51.710822 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:51.710794 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-5cd48b54dd-8tn7q" Oct 30 18:05:51.992185 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:51.992103 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8d1062-595d67866c-92d2c"] Oct 30 18:05:52.021065 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:52.021032 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-455086-6c947b47-fllqz" podUID="dfcedd36-934a-4257-b1e9-db368ed2bcd5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:05:52.717653 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:52.717627 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-6dcd6678f9-jjjkd" Oct 30 18:05:53.675617 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:53.675583 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:05:53.675792 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:53.675638 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" podUID="b003d430-64cd-4a08-9a96-6b326a534b5e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:05:55.758428 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.758364 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 18:05:55.823520 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.823486 4965 generic.go:356] "Generic (PLEG): container finished" podID="b003d430-64cd-4a08-9a96-6b326a534b5e" containerID="efb393c33e8a75e090b7e950d53b95bd27c15cf059e5ca90be6ad2119efd36f6" exitCode=0 Oct 30 18:05:55.823699 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.823537 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" Oct 30 18:05:55.823699 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.823555 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" event={"ID":"b003d430-64cd-4a08-9a96-6b326a534b5e","Type":"ContainerDied","Data":"efb393c33e8a75e090b7e950d53b95bd27c15cf059e5ca90be6ad2119efd36f6"} Oct 30 18:05:55.823699 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.823659 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k" event={"ID":"b003d430-64cd-4a08-9a96-6b326a534b5e","Type":"ContainerDied","Data":"164007abc82072fe9f1f3fb7b2718671783225bb4eca6ce36394e87979df65e1"} Oct 30 18:05:55.823699 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.823672 4965 scope.go:122] "RemoveContainer" containerID="efb393c33e8a75e090b7e950d53b95bd27c15cf059e5ca90be6ad2119efd36f6" Oct 30 18:05:55.835304 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.835283 4965 scope.go:122] "RemoveContainer" containerID="89c618517e9de02d4828e6dfbb050769a890b36188c3694d68cb8b73e90f163f" Oct 30 18:05:55.846736 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.846703 4965 scope.go:122] "RemoveContainer" containerID="efb393c33e8a75e090b7e950d53b95bd27c15cf059e5ca90be6ad2119efd36f6" Oct 30 18:05:55.847397 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:05:55.847373 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efb393c33e8a75e090b7e950d53b95bd27c15cf059e5ca90be6ad2119efd36f6\": container with ID starting with efb393c33e8a75e090b7e950d53b95bd27c15cf059e5ca90be6ad2119efd36f6 not found: ID does not exist" containerID="efb393c33e8a75e090b7e950d53b95bd27c15cf059e5ca90be6ad2119efd36f6" Oct 30 18:05:55.847509 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.847421 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efb393c33e8a75e090b7e950d53b95bd27c15cf059e5ca90be6ad2119efd36f6"} err="failed to get container status \"efb393c33e8a75e090b7e950d53b95bd27c15cf059e5ca90be6ad2119efd36f6\": rpc error: code = NotFound desc = could not find container \"efb393c33e8a75e090b7e950d53b95bd27c15cf059e5ca90be6ad2119efd36f6\": container with ID starting with efb393c33e8a75e090b7e950d53b95bd27c15cf059e5ca90be6ad2119efd36f6 not found: ID does not exist" Oct 30 18:05:55.847509 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.847446 4965 scope.go:122] "RemoveContainer" containerID="89c618517e9de02d4828e6dfbb050769a890b36188c3694d68cb8b73e90f163f" Oct 30 18:05:55.848722 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:05:55.848694 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89c618517e9de02d4828e6dfbb050769a890b36188c3694d68cb8b73e90f163f\": container with ID starting with 89c618517e9de02d4828e6dfbb050769a890b36188c3694d68cb8b73e90f163f not found: ID does not exist" containerID="89c618517e9de02d4828e6dfbb050769a890b36188c3694d68cb8b73e90f163f" Oct 30 18:05:55.848820 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.848741 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89c618517e9de02d4828e6dfbb050769a890b36188c3694d68cb8b73e90f163f"} err="failed to get container status \"89c618517e9de02d4828e6dfbb050769a890b36188c3694d68cb8b73e90f163f\": rpc error: code = NotFound desc = could not find container \"89c618517e9de02d4828e6dfbb050769a890b36188c3694d68cb8b73e90f163f\": container with ID starting with 89c618517e9de02d4828e6dfbb050769a890b36188c3694d68cb8b73e90f163f not found: ID does not exist" Oct 30 18:05:55.862241 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.862210 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b003d430-64cd-4a08-9a96-6b326a534b5e-kube-api-access-r4npr\" (UniqueName: \"kubernetes.io/projected/b003d430-64cd-4a08-9a96-6b326a534b5e-kube-api-access-r4npr\") pod \"b003d430-64cd-4a08-9a96-6b326a534b5e\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " Oct 30 18:05:55.862365 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.862278 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-istio-envoy\") pod \"b003d430-64cd-4a08-9a96-6b326a534b5e\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " Oct 30 18:05:55.862365 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.862336 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-workload-socket\") pod \"b003d430-64cd-4a08-9a96-6b326a534b5e\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " Oct 30 18:05:55.862492 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.862460 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b003d430-64cd-4a08-9a96-6b326a534b5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b003d430-64cd-4a08-9a96-6b326a534b5e-istiod-ca-cert\") pod \"b003d430-64cd-4a08-9a96-6b326a534b5e\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " Oct 30 18:05:55.862552 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.862537 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b003d430-64cd-4a08-9a96-6b326a534b5e-istio-token\" (UniqueName: \"kubernetes.io/projected/b003d430-64cd-4a08-9a96-6b326a534b5e-istio-token\") pod \"b003d430-64cd-4a08-9a96-6b326a534b5e\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " Oct 30 18:05:55.862679 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.862651 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-istio-data\") pod \"b003d430-64cd-4a08-9a96-6b326a534b5e\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " Oct 30 18:05:55.862750 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.862682 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b003d430-64cd-4a08-9a96-6b326a534b5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b003d430-64cd-4a08-9a96-6b326a534b5e-istio-podinfo\") pod \"b003d430-64cd-4a08-9a96-6b326a534b5e\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " Oct 30 18:05:55.862750 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.862713 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-workload-certs\") pod \"b003d430-64cd-4a08-9a96-6b326a534b5e\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " Oct 30 18:05:55.862750 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.862737 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-credential-socket\") pod \"b003d430-64cd-4a08-9a96-6b326a534b5e\" (UID: \"b003d430-64cd-4a08-9a96-6b326a534b5e\") " Oct 30 18:05:55.862940 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.862755 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-workload-socket" pod "b003d430-64cd-4a08-9a96-6b326a534b5e" (UID: "b003d430-64cd-4a08-9a96-6b326a534b5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:55.863832 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.863215 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-istio-data" pod "b003d430-64cd-4a08-9a96-6b326a534b5e" (UID: "b003d430-64cd-4a08-9a96-6b326a534b5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:55.864007 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.863666 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-workload-certs" pod "b003d430-64cd-4a08-9a96-6b326a534b5e" (UID: "b003d430-64cd-4a08-9a96-6b326a534b5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:55.864120 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.863680 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b003d430-64cd-4a08-9a96-6b326a534b5e-istiod-ca-cert" pod "b003d430-64cd-4a08-9a96-6b326a534b5e" (UID: "b003d430-64cd-4a08-9a96-6b326a534b5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:05:55.864222 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.863840 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-credential-socket" pod "b003d430-64cd-4a08-9a96-6b326a534b5e" (UID: "b003d430-64cd-4a08-9a96-6b326a534b5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:55.864330 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.863288 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:55.866107 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.866087 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-istio-envoy" pod "b003d430-64cd-4a08-9a96-6b326a534b5e" (UID: "b003d430-64cd-4a08-9a96-6b326a534b5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:55.867718 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.867682 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b003d430-64cd-4a08-9a96-6b326a534b5e-istio-token" pod "b003d430-64cd-4a08-9a96-6b326a534b5e" (UID: "b003d430-64cd-4a08-9a96-6b326a534b5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:55.868041 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.867993 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b003d430-64cd-4a08-9a96-6b326a534b5e-istio-podinfo" pod "b003d430-64cd-4a08-9a96-6b326a534b5e" (UID: "b003d430-64cd-4a08-9a96-6b326a534b5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:05:55.868145 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.868034 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b003d430-64cd-4a08-9a96-6b326a534b5e-kube-api-access-r4npr" pod "b003d430-64cd-4a08-9a96-6b326a534b5e" (UID: "b003d430-64cd-4a08-9a96-6b326a534b5e"). InnerVolumeSpecName "kube-api-access-r4npr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:55.965515 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.965479 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:55.965515 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.965509 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b003d430-64cd-4a08-9a96-6b326a534b5e-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:55.965515 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.965519 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:55.965758 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.965527 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:55.965758 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.965537 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4npr\" (UniqueName: \"kubernetes.io/projected/b003d430-64cd-4a08-9a96-6b326a534b5e-kube-api-access-r4npr\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:55.965758 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.965546 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b003d430-64cd-4a08-9a96-6b326a534b5e-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:55.965758 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.965554 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b003d430-64cd-4a08-9a96-6b326a534b5e-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:55.965758 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:55.965563 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b003d430-64cd-4a08-9a96-6b326a534b5e-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:05:56.177169 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:56.177136 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k"] Oct 30 18:05:56.187188 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:56.187160 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a17354-openshift-default-67fb566dc7-vm28k"] Oct 30 18:05:56.992544 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:56.992505 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8d1062-595d67866c-92d2c" podUID="1a68dfa4-44fa-40d0-bf51-ecac52a252d7" containerName="payload-pre-processing" containerID="cri-o://6ae936b4b049bef952eb1349149bcbfa65ffe5c99b8093ef79a93d70f8be7d62" gracePeriod=25 Oct 30 18:05:57.268063 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:57.267979 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8d1062-595d67866c-92d2c" podUID="1a68dfa4-44fa-40d0-bf51-ecac52a252d7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:05:57.414189 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:57.414152 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-d207f8-d8677dc5d-5pgjt" podUID="6706cbd9-edfa-4b4d-ae4a-44a037e61efc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:05:58.099113 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:05:58.099080 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b003d430-64cd-4a08-9a96-6b326a534b5e" path="/var/lib/kubelet/pods/b003d430-64cd-4a08-9a96-6b326a534b5e/volumes" Oct 30 18:06:00.233828 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:00.233793 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp"] Oct 30 18:06:00.234355 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:00.234075 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" podUID="306a7985-a3c4-46d3-a640-5663ace6b620" containerName="istio-proxy" containerID="cri-o://0b5949d4be3e560ebe13e9a3048c66b5b94c5d558e2da9b444c02a9e0c5a7a41" gracePeriod=30 Oct 30 18:06:01.290838 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:01.290796 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-016f1d-78b77d5dc4-qxvr6"] Oct 30 18:06:01.325182 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:01.325148 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016f1d-5d984ddf9-sjb6b"] Oct 30 18:06:02.021182 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:02.021141 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-455086-6c947b47-fllqz" podUID="dfcedd36-934a-4257-b1e9-db368ed2bcd5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:06:05.216381 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.216316 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-455086-6c947b47-fllqz" Oct 30 18:06:05.320376 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.320348 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfcedd36-934a-4257-b1e9-db368ed2bcd5-kube-api-access-8jlrn\" (UniqueName: \"kubernetes.io/projected/dfcedd36-934a-4257-b1e9-db368ed2bcd5-kube-api-access-8jlrn\") pod \"dfcedd36-934a-4257-b1e9-db368ed2bcd5\" (UID: \"dfcedd36-934a-4257-b1e9-db368ed2bcd5\") " Oct 30 18:06:05.320578 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.320563 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfcedd36-934a-4257-b1e9-db368ed2bcd5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfcedd36-934a-4257-b1e9-db368ed2bcd5-plugins-config-volume\") pod \"dfcedd36-934a-4257-b1e9-db368ed2bcd5\" (UID: \"dfcedd36-934a-4257-b1e9-db368ed2bcd5\") " Oct 30 18:06:05.320952 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.320922 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfcedd36-934a-4257-b1e9-db368ed2bcd5-plugins-config-volume" pod "dfcedd36-934a-4257-b1e9-db368ed2bcd5" (UID: "dfcedd36-934a-4257-b1e9-db368ed2bcd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:06:05.322467 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.322430 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfcedd36-934a-4257-b1e9-db368ed2bcd5-kube-api-access-8jlrn" pod "dfcedd36-934a-4257-b1e9-db368ed2bcd5" (UID: "dfcedd36-934a-4257-b1e9-db368ed2bcd5"). InnerVolumeSpecName "kube-api-access-8jlrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:05.424147 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.424113 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfcedd36-934a-4257-b1e9-db368ed2bcd5-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:05.424147 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.424146 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jlrn\" (UniqueName: \"kubernetes.io/projected/dfcedd36-934a-4257-b1e9-db368ed2bcd5-kube-api-access-8jlrn\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:05.479016 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.478988 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:06:05.625532 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.625502 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-workload-certs\") pod \"306a7985-a3c4-46d3-a640-5663ace6b620\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " Oct 30 18:06:05.625711 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.625560 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-credential-socket\") pod \"306a7985-a3c4-46d3-a640-5663ace6b620\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " Oct 30 18:06:05.625711 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.625612 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-istio-envoy\") pod \"306a7985-a3c4-46d3-a640-5663ace6b620\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " Oct 30 18:06:05.625711 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.625682 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/306a7985-a3c4-46d3-a640-5663ace6b620-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306a7985-a3c4-46d3-a640-5663ace6b620-istiod-ca-cert\") pod \"306a7985-a3c4-46d3-a640-5663ace6b620\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " Oct 30 18:06:05.625921 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.625713 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/306a7985-a3c4-46d3-a640-5663ace6b620-kube-api-access-fdmzd\" (UniqueName: \"kubernetes.io/projected/306a7985-a3c4-46d3-a640-5663ace6b620-kube-api-access-fdmzd\") pod \"306a7985-a3c4-46d3-a640-5663ace6b620\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " Oct 30 18:06:05.625921 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.625742 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-istio-data\") pod \"306a7985-a3c4-46d3-a640-5663ace6b620\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " Oct 30 18:06:05.625921 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.625767 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/306a7985-a3c4-46d3-a640-5663ace6b620-istio-token\" (UniqueName: \"kubernetes.io/projected/306a7985-a3c4-46d3-a640-5663ace6b620-istio-token\") pod \"306a7985-a3c4-46d3-a640-5663ace6b620\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " Oct 30 18:06:05.625921 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.625784 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-workload-certs" pod "306a7985-a3c4-46d3-a640-5663ace6b620" (UID: "306a7985-a3c4-46d3-a640-5663ace6b620"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:05.625921 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.625893 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-workload-socket\") pod \"306a7985-a3c4-46d3-a640-5663ace6b620\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " Oct 30 18:06:05.625921 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.625857 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-credential-socket" pod "306a7985-a3c4-46d3-a640-5663ace6b620" (UID: "306a7985-a3c4-46d3-a640-5663ace6b620"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:05.626272 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.625996 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/306a7985-a3c4-46d3-a640-5663ace6b620-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306a7985-a3c4-46d3-a640-5663ace6b620-istio-podinfo\") pod \"306a7985-a3c4-46d3-a640-5663ace6b620\" (UID: \"306a7985-a3c4-46d3-a640-5663ace6b620\") " Oct 30 18:06:05.626272 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.626087 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-istio-data" pod "306a7985-a3c4-46d3-a640-5663ace6b620" (UID: "306a7985-a3c4-46d3-a640-5663ace6b620"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:05.626272 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.626122 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-workload-socket" pod "306a7985-a3c4-46d3-a640-5663ace6b620" (UID: "306a7985-a3c4-46d3-a640-5663ace6b620"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:05.626272 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.626165 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/306a7985-a3c4-46d3-a640-5663ace6b620-istiod-ca-cert" pod "306a7985-a3c4-46d3-a640-5663ace6b620" (UID: "306a7985-a3c4-46d3-a640-5663ace6b620"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:06:05.626567 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.626548 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:05.626631 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.626569 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:05.626631 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.626579 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306a7985-a3c4-46d3-a640-5663ace6b620-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:05.626631 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.626589 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:05.626631 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.626596 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:05.628417 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.628392 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-istio-envoy" pod "306a7985-a3c4-46d3-a640-5663ace6b620" (UID: "306a7985-a3c4-46d3-a640-5663ace6b620"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:05.628417 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.628413 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/306a7985-a3c4-46d3-a640-5663ace6b620-kube-api-access-fdmzd" pod "306a7985-a3c4-46d3-a640-5663ace6b620" (UID: "306a7985-a3c4-46d3-a640-5663ace6b620"). InnerVolumeSpecName "kube-api-access-fdmzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:05.628605 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.628590 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/306a7985-a3c4-46d3-a640-5663ace6b620-istio-podinfo" pod "306a7985-a3c4-46d3-a640-5663ace6b620" (UID: "306a7985-a3c4-46d3-a640-5663ace6b620"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:06:05.628704 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.628688 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/306a7985-a3c4-46d3-a640-5663ace6b620-istio-token" pod "306a7985-a3c4-46d3-a640-5663ace6b620" (UID: "306a7985-a3c4-46d3-a640-5663ace6b620"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:05.724691 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.724605 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll"] Oct 30 18:06:05.728082 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.728057 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdmzd\" (UniqueName: \"kubernetes.io/projected/306a7985-a3c4-46d3-a640-5663ace6b620-kube-api-access-fdmzd\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:05.728082 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.728082 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/306a7985-a3c4-46d3-a640-5663ace6b620-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:05.728232 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.728092 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306a7985-a3c4-46d3-a640-5663ace6b620-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:05.728232 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.728102 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306a7985-a3c4-46d3-a640-5663ace6b620-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:05.947758 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.947722 4965 generic.go:356] "Generic (PLEG): container finished" podID="dfcedd36-934a-4257-b1e9-db368ed2bcd5" containerID="1a95329f71d22d5c904d568dae3ca624bb25ed85e95cf68a0ed5eba499cadb72" exitCode=0 Oct 30 18:06:05.947758 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.947761 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-455086-6c947b47-fllqz" Oct 30 18:06:05.948018 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.947797 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-455086-6c947b47-fllqz" event={"ID":"dfcedd36-934a-4257-b1e9-db368ed2bcd5","Type":"ContainerDied","Data":"1a95329f71d22d5c904d568dae3ca624bb25ed85e95cf68a0ed5eba499cadb72"} Oct 30 18:06:05.948018 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.947850 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-455086-6c947b47-fllqz" event={"ID":"dfcedd36-934a-4257-b1e9-db368ed2bcd5","Type":"ContainerDied","Data":"72422bec88f7b802e9b33dc5d647f0281af5eb05c420b887466c4810318ec103"} Oct 30 18:06:05.948018 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.947917 4965 scope.go:122] "RemoveContainer" containerID="1a95329f71d22d5c904d568dae3ca624bb25ed85e95cf68a0ed5eba499cadb72" Oct 30 18:06:05.949946 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.949493 4965 generic.go:356] "Generic (PLEG): container finished" podID="306a7985-a3c4-46d3-a640-5663ace6b620" containerID="0b5949d4be3e560ebe13e9a3048c66b5b94c5d558e2da9b444c02a9e0c5a7a41" exitCode=0 Oct 30 18:06:05.949946 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.949543 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" Oct 30 18:06:05.949946 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.949580 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" event={"ID":"306a7985-a3c4-46d3-a640-5663ace6b620","Type":"ContainerDied","Data":"0b5949d4be3e560ebe13e9a3048c66b5b94c5d558e2da9b444c02a9e0c5a7a41"} Oct 30 18:06:05.949946 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.949625 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp" event={"ID":"306a7985-a3c4-46d3-a640-5663ace6b620","Type":"ContainerDied","Data":"5010cb16392103ac2864b952bb4e79e964988f69594f70c3a608513059bd5264"} Oct 30 18:06:05.959501 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.959478 4965 scope.go:122] "RemoveContainer" containerID="1a95329f71d22d5c904d568dae3ca624bb25ed85e95cf68a0ed5eba499cadb72" Oct 30 18:06:05.959932 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:06:05.959818 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a95329f71d22d5c904d568dae3ca624bb25ed85e95cf68a0ed5eba499cadb72\": container with ID starting with 1a95329f71d22d5c904d568dae3ca624bb25ed85e95cf68a0ed5eba499cadb72 not found: ID does not exist" containerID="1a95329f71d22d5c904d568dae3ca624bb25ed85e95cf68a0ed5eba499cadb72" Oct 30 18:06:05.959932 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.959847 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a95329f71d22d5c904d568dae3ca624bb25ed85e95cf68a0ed5eba499cadb72"} err="failed to get container status \"1a95329f71d22d5c904d568dae3ca624bb25ed85e95cf68a0ed5eba499cadb72\": rpc error: code = NotFound desc = could not find container \"1a95329f71d22d5c904d568dae3ca624bb25ed85e95cf68a0ed5eba499cadb72\": container with ID starting with 1a95329f71d22d5c904d568dae3ca624bb25ed85e95cf68a0ed5eba499cadb72 not found: ID does not exist" Oct 30 18:06:05.959932 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.959906 4965 scope.go:122] "RemoveContainer" containerID="0b5949d4be3e560ebe13e9a3048c66b5b94c5d558e2da9b444c02a9e0c5a7a41" Oct 30 18:06:05.989683 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.989664 4965 scope.go:122] "RemoveContainer" containerID="0b5949d4be3e560ebe13e9a3048c66b5b94c5d558e2da9b444c02a9e0c5a7a41" Oct 30 18:06:05.990008 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:06:05.989980 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b5949d4be3e560ebe13e9a3048c66b5b94c5d558e2da9b444c02a9e0c5a7a41\": container with ID starting with 0b5949d4be3e560ebe13e9a3048c66b5b94c5d558e2da9b444c02a9e0c5a7a41 not found: ID does not exist" containerID="0b5949d4be3e560ebe13e9a3048c66b5b94c5d558e2da9b444c02a9e0c5a7a41" Oct 30 18:06:05.990082 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:05.990011 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b5949d4be3e560ebe13e9a3048c66b5b94c5d558e2da9b444c02a9e0c5a7a41"} err="failed to get container status \"0b5949d4be3e560ebe13e9a3048c66b5b94c5d558e2da9b444c02a9e0c5a7a41\": rpc error: code = NotFound desc = could not find container \"0b5949d4be3e560ebe13e9a3048c66b5b94c5d558e2da9b444c02a9e0c5a7a41\": container with ID starting with 0b5949d4be3e560ebe13e9a3048c66b5b94c5d558e2da9b444c02a9e0c5a7a41 not found: ID does not exist" Oct 30 18:06:06.000268 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:06:06.000235 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:06:06.004274 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.004249 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-455086-6c947b47-fllqz"] Oct 30 18:06:06.011360 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.011336 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-455086-6c947b47-fllqz"] Oct 30 18:06:06.043069 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.043045 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp"] Oct 30 18:06:06.048457 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.048432 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-q6bzp"] Oct 30 18:06:06.099345 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.099313 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="306a7985-a3c4-46d3-a640-5663ace6b620" path="/var/lib/kubelet/pods/306a7985-a3c4-46d3-a640-5663ace6b620/volumes" Oct 30 18:06:06.099841 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.099824 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfcedd36-934a-4257-b1e9-db368ed2bcd5" path="/var/lib/kubelet/pods/dfcedd36-934a-4257-b1e9-db368ed2bcd5/volumes" Oct 30 18:06:06.291475 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.291360 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016f1d-78b77d5dc4-qxvr6" podUID="17d99e83-45e9-446a-9525-ee96fe9ff19a" containerName="payload-processing" containerID="cri-o://46445a17a3acec951ad67c73c3b719c4b0ef223362685f63f01eb1717020fcd1" gracePeriod=25 Oct 30 18:06:06.325442 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.325412 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016f1d-5d984ddf9-sjb6b" podUID="84fbd1da-e017-490a-8db0-1c9555ceb3b4" containerName="payload-pre-processing" containerID="cri-o://36b2412a074d7f4bb195cd58ed4464844915709c73526c7bc87d83b910c6b3ae" gracePeriod=25 Oct 30 18:06:06.614219 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.614189 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016f1d-5d984ddf9-sjb6b" Oct 30 18:06:06.630154 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.630130 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016f1d-78b77d5dc4-qxvr6" Oct 30 18:06:06.742544 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.742507 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17d99e83-45e9-446a-9525-ee96fe9ff19a-kube-api-access-qz685\" (UniqueName: \"kubernetes.io/projected/17d99e83-45e9-446a-9525-ee96fe9ff19a-kube-api-access-qz685\") pod \"17d99e83-45e9-446a-9525-ee96fe9ff19a\" (UID: \"17d99e83-45e9-446a-9525-ee96fe9ff19a\") " Oct 30 18:06:06.742725 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.742645 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84fbd1da-e017-490a-8db0-1c9555ceb3b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84fbd1da-e017-490a-8db0-1c9555ceb3b4-plugins-config-volume\") pod \"84fbd1da-e017-490a-8db0-1c9555ceb3b4\" (UID: \"84fbd1da-e017-490a-8db0-1c9555ceb3b4\") " Oct 30 18:06:06.742725 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.742672 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17d99e83-45e9-446a-9525-ee96fe9ff19a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17d99e83-45e9-446a-9525-ee96fe9ff19a-plugins-config-volume\") pod \"17d99e83-45e9-446a-9525-ee96fe9ff19a\" (UID: \"17d99e83-45e9-446a-9525-ee96fe9ff19a\") " Oct 30 18:06:06.742725 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.742719 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84fbd1da-e017-490a-8db0-1c9555ceb3b4-kube-api-access-xksz4\" (UniqueName: \"kubernetes.io/projected/84fbd1da-e017-490a-8db0-1c9555ceb3b4-kube-api-access-xksz4\") pod \"84fbd1da-e017-490a-8db0-1c9555ceb3b4\" (UID: \"84fbd1da-e017-490a-8db0-1c9555ceb3b4\") " Oct 30 18:06:06.743131 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.743104 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17d99e83-45e9-446a-9525-ee96fe9ff19a-plugins-config-volume" pod "17d99e83-45e9-446a-9525-ee96fe9ff19a" (UID: "17d99e83-45e9-446a-9525-ee96fe9ff19a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:06:06.743282 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.743129 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84fbd1da-e017-490a-8db0-1c9555ceb3b4-plugins-config-volume" pod "84fbd1da-e017-490a-8db0-1c9555ceb3b4" (UID: "84fbd1da-e017-490a-8db0-1c9555ceb3b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:06:06.743282 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.743267 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84fbd1da-e017-490a-8db0-1c9555ceb3b4-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:06.743376 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.743289 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17d99e83-45e9-446a-9525-ee96fe9ff19a-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:06.744646 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.744625 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17d99e83-45e9-446a-9525-ee96fe9ff19a-kube-api-access-qz685" pod "17d99e83-45e9-446a-9525-ee96fe9ff19a" (UID: "17d99e83-45e9-446a-9525-ee96fe9ff19a"). InnerVolumeSpecName "kube-api-access-qz685". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:06.744752 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.744733 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84fbd1da-e017-490a-8db0-1c9555ceb3b4-kube-api-access-xksz4" pod "84fbd1da-e017-490a-8db0-1c9555ceb3b4" (UID: "84fbd1da-e017-490a-8db0-1c9555ceb3b4"). InnerVolumeSpecName "kube-api-access-xksz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:06.844567 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.844487 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xksz4\" (UniqueName: \"kubernetes.io/projected/84fbd1da-e017-490a-8db0-1c9555ceb3b4-kube-api-access-xksz4\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:06.844567 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.844516 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz685\" (UniqueName: \"kubernetes.io/projected/17d99e83-45e9-446a-9525-ee96fe9ff19a-kube-api-access-qz685\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:06.960839 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.958619 4965 generic.go:356] "Generic (PLEG): container finished" podID="84fbd1da-e017-490a-8db0-1c9555ceb3b4" containerID="36b2412a074d7f4bb195cd58ed4464844915709c73526c7bc87d83b910c6b3ae" exitCode=0 Oct 30 18:06:06.960839 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.958852 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016f1d-5d984ddf9-sjb6b" event={"ID":"84fbd1da-e017-490a-8db0-1c9555ceb3b4","Type":"ContainerDied","Data":"36b2412a074d7f4bb195cd58ed4464844915709c73526c7bc87d83b910c6b3ae"} Oct 30 18:06:06.960839 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.958997 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016f1d-5d984ddf9-sjb6b" event={"ID":"84fbd1da-e017-490a-8db0-1c9555ceb3b4","Type":"ContainerDied","Data":"b7581708f371f2316637328221ae82ce23e5d379a6a1c4edd47ffd1b3862e4ac"} Oct 30 18:06:06.960839 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.959040 4965 scope.go:122] "RemoveContainer" containerID="36b2412a074d7f4bb195cd58ed4464844915709c73526c7bc87d83b910c6b3ae" Oct 30 18:06:06.960839 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.959236 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016f1d-5d984ddf9-sjb6b" Oct 30 18:06:06.971957 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.971921 4965 generic.go:356] "Generic (PLEG): container finished" podID="17d99e83-45e9-446a-9525-ee96fe9ff19a" containerID="46445a17a3acec951ad67c73c3b719c4b0ef223362685f63f01eb1717020fcd1" exitCode=0 Oct 30 18:06:06.972108 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.972000 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016f1d-78b77d5dc4-qxvr6" Oct 30 18:06:06.972108 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.972006 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016f1d-78b77d5dc4-qxvr6" event={"ID":"17d99e83-45e9-446a-9525-ee96fe9ff19a","Type":"ContainerDied","Data":"46445a17a3acec951ad67c73c3b719c4b0ef223362685f63f01eb1717020fcd1"} Oct 30 18:06:06.972108 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.972067 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-016f1d-78b77d5dc4-qxvr6" event={"ID":"17d99e83-45e9-446a-9525-ee96fe9ff19a","Type":"ContainerDied","Data":"3df6f725988dd7af3eb2890341090f1c3aa2b3aa18cf7befb5299c2159b9e1cf"} Oct 30 18:06:06.991200 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.991182 4965 scope.go:122] "RemoveContainer" containerID="36b2412a074d7f4bb195cd58ed4464844915709c73526c7bc87d83b910c6b3ae" Oct 30 18:06:06.991469 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:06:06.991449 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36b2412a074d7f4bb195cd58ed4464844915709c73526c7bc87d83b910c6b3ae\": container with ID starting with 36b2412a074d7f4bb195cd58ed4464844915709c73526c7bc87d83b910c6b3ae not found: ID does not exist" containerID="36b2412a074d7f4bb195cd58ed4464844915709c73526c7bc87d83b910c6b3ae" Oct 30 18:06:06.991555 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.991479 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36b2412a074d7f4bb195cd58ed4464844915709c73526c7bc87d83b910c6b3ae"} err="failed to get container status \"36b2412a074d7f4bb195cd58ed4464844915709c73526c7bc87d83b910c6b3ae\": rpc error: code = NotFound desc = could not find container \"36b2412a074d7f4bb195cd58ed4464844915709c73526c7bc87d83b910c6b3ae\": container with ID starting with 36b2412a074d7f4bb195cd58ed4464844915709c73526c7bc87d83b910c6b3ae not found: ID does not exist" Oct 30 18:06:06.991555 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:06.991502 4965 scope.go:122] "RemoveContainer" containerID="46445a17a3acec951ad67c73c3b719c4b0ef223362685f63f01eb1717020fcd1" Oct 30 18:06:07.000407 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:07.000385 4965 scope.go:122] "RemoveContainer" containerID="46445a17a3acec951ad67c73c3b719c4b0ef223362685f63f01eb1717020fcd1" Oct 30 18:06:07.000665 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:06:07.000646 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46445a17a3acec951ad67c73c3b719c4b0ef223362685f63f01eb1717020fcd1\": container with ID starting with 46445a17a3acec951ad67c73c3b719c4b0ef223362685f63f01eb1717020fcd1 not found: ID does not exist" containerID="46445a17a3acec951ad67c73c3b719c4b0ef223362685f63f01eb1717020fcd1" Oct 30 18:06:07.000720 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:07.000670 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46445a17a3acec951ad67c73c3b719c4b0ef223362685f63f01eb1717020fcd1"} err="failed to get container status \"46445a17a3acec951ad67c73c3b719c4b0ef223362685f63f01eb1717020fcd1\": rpc error: code = NotFound desc = could not find container \"46445a17a3acec951ad67c73c3b719c4b0ef223362685f63f01eb1717020fcd1\": container with ID starting with 46445a17a3acec951ad67c73c3b719c4b0ef223362685f63f01eb1717020fcd1 not found: ID does not exist" Oct 30 18:06:07.038894 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:07.036625 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-016f1d-78b77d5dc4-qxvr6"] Oct 30 18:06:07.043211 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:07.043174 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-016f1d-78b77d5dc4-qxvr6"] Oct 30 18:06:07.077050 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:07.077019 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016f1d-5d984ddf9-sjb6b"] Oct 30 18:06:07.080806 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:07.080781 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-016f1d-5d984ddf9-sjb6b"] Oct 30 18:06:07.268213 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:07.268124 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8d1062-595d67866c-92d2c" podUID="1a68dfa4-44fa-40d0-bf51-ecac52a252d7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:06:07.414170 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:07.414133 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-d207f8-d8677dc5d-5pgjt" podUID="6706cbd9-edfa-4b4d-ae4a-44a037e61efc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:06:08.099163 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:08.099128 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17d99e83-45e9-446a-9525-ee96fe9ff19a" path="/var/lib/kubelet/pods/17d99e83-45e9-446a-9525-ee96fe9ff19a/volumes" Oct 30 18:06:08.099591 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:08.099576 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84fbd1da-e017-490a-8db0-1c9555ceb3b4" path="/var/lib/kubelet/pods/84fbd1da-e017-490a-8db0-1c9555ceb3b4/volumes" Oct 30 18:06:08.897299 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:08.897259 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll"] Oct 30 18:06:08.897653 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:08.897510 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" podUID="f170ecd7-a0bf-469a-b3ac-cbd739327d4d" containerName="istio-proxy" containerID="cri-o://4a6d4e5b44446802b627b278445a8fb338e3d490ce429967110336398727914b" gracePeriod=30 Oct 30 18:06:09.047944 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:09.023295 4965 generic.go:356] "Generic (PLEG): container finished" podID="6706cbd9-edfa-4b4d-ae4a-44a037e61efc" containerID="213ec240dba617b0f3d51ad3b068504c54a851a62bf770f48a19e5c665cf5794" exitCode=0 Oct 30 18:06:09.047944 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:09.023460 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-d207f8-d8677dc5d-5pgjt" event={"ID":"6706cbd9-edfa-4b4d-ae4a-44a037e61efc","Type":"ContainerDied","Data":"213ec240dba617b0f3d51ad3b068504c54a851a62bf770f48a19e5c665cf5794"} Oct 30 18:06:09.103977 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:09.103951 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-d207f8-d8677dc5d-5pgjt" Oct 30 18:06:09.177290 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:09.177259 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6706cbd9-edfa-4b4d-ae4a-44a037e61efc-kube-api-access-nszdr\" (UniqueName: \"kubernetes.io/projected/6706cbd9-edfa-4b4d-ae4a-44a037e61efc-kube-api-access-nszdr\") pod \"6706cbd9-edfa-4b4d-ae4a-44a037e61efc\" (UID: \"6706cbd9-edfa-4b4d-ae4a-44a037e61efc\") " Oct 30 18:06:09.177490 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:09.177339 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6706cbd9-edfa-4b4d-ae4a-44a037e61efc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6706cbd9-edfa-4b4d-ae4a-44a037e61efc-plugins-config-volume\") pod \"6706cbd9-edfa-4b4d-ae4a-44a037e61efc\" (UID: \"6706cbd9-edfa-4b4d-ae4a-44a037e61efc\") " Oct 30 18:06:09.177758 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:09.177733 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6706cbd9-edfa-4b4d-ae4a-44a037e61efc-plugins-config-volume" pod "6706cbd9-edfa-4b4d-ae4a-44a037e61efc" (UID: "6706cbd9-edfa-4b4d-ae4a-44a037e61efc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:06:09.177930 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:09.177914 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6706cbd9-edfa-4b4d-ae4a-44a037e61efc-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:09.179498 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:09.179471 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6706cbd9-edfa-4b4d-ae4a-44a037e61efc-kube-api-access-nszdr" pod "6706cbd9-edfa-4b4d-ae4a-44a037e61efc" (UID: "6706cbd9-edfa-4b4d-ae4a-44a037e61efc"). InnerVolumeSpecName "kube-api-access-nszdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:09.280807 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:09.279810 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nszdr\" (UniqueName: \"kubernetes.io/projected/6706cbd9-edfa-4b4d-ae4a-44a037e61efc-kube-api-access-nszdr\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:10.031654 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:10.031612 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-d207f8-d8677dc5d-5pgjt" Oct 30 18:06:10.031654 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:10.031633 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-d207f8-d8677dc5d-5pgjt" event={"ID":"6706cbd9-edfa-4b4d-ae4a-44a037e61efc","Type":"ContainerDied","Data":"5d20c92066c27f09f219fb15ac671043a27f4d4e6af16c70e29bf26815210252"} Oct 30 18:06:10.032240 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:10.031700 4965 scope.go:122] "RemoveContainer" containerID="213ec240dba617b0f3d51ad3b068504c54a851a62bf770f48a19e5c665cf5794" Oct 30 18:06:10.085066 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:10.085035 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-d207f8-d8677dc5d-5pgjt"] Oct 30 18:06:10.098976 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:10.098948 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-d207f8-d8677dc5d-5pgjt"] Oct 30 18:06:12.106454 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:12.106420 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6706cbd9-edfa-4b4d-ae4a-44a037e61efc" path="/var/lib/kubelet/pods/6706cbd9-edfa-4b4d-ae4a-44a037e61efc/volumes" Oct 30 18:06:13.874559 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:13.874522 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m"] Oct 30 18:06:13.875177 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:13.874839 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" podUID="82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e" containerName="istio-proxy" containerID="cri-o://f298491a0f8a7f978b59eca8e56c6796ee878ba49feb6f86491d7a6834ebe302" gracePeriod=30 Oct 30 18:06:14.094699 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:14.094666 4965 generic.go:356] "Generic (PLEG): container finished" podID="f170ecd7-a0bf-469a-b3ac-cbd739327d4d" containerID="4a6d4e5b44446802b627b278445a8fb338e3d490ce429967110336398727914b" exitCode=0 Oct 30 18:06:14.101006 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:14.100976 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" event={"ID":"f170ecd7-a0bf-469a-b3ac-cbd739327d4d","Type":"ContainerDied","Data":"4a6d4e5b44446802b627b278445a8fb338e3d490ce429967110336398727914b"} Oct 30 18:06:14.213569 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:14.213542 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 18:06:14.348116 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:14.348080 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istio-token\" (UniqueName: \"kubernetes.io/projected/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istio-token\") pod \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " Oct 30 18:06:14.348116 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:14.348118 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-kube-api-access-k7g7x\" (UniqueName: \"kubernetes.io/projected/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-kube-api-access-k7g7x\") pod \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " Oct 30 18:06:14.348383 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:14.348163 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-workload-certs\") pod \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " Oct 30 18:06:14.348383 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:14.348197 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-credential-socket\") pod \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " Oct 30 18:06:14.348383 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:14.348283 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istio-podinfo\") pod \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " Oct 30 18:06:14.348383 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:14.348332 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istio-data\") pod \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " Oct 30 18:06:14.348383 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:14.348364 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istiod-ca-cert\") pod \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " Oct 30 18:06:14.348639 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:14.348389 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istio-envoy\") pod \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " Oct 30 18:06:14.348639 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:14.348582 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-credential-socket" pod "f170ecd7-a0bf-469a-b3ac-cbd739327d4d" (UID: "f170ecd7-a0bf-469a-b3ac-cbd739327d4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:14.348639 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:14.348599 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-workload-socket\") pod \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\" (UID: \"f170ecd7-a0bf-469a-b3ac-cbd739327d4d\") " Oct 30 18:06:14.348811 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:14.348709 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-workload-certs" pod "f170ecd7-a0bf-469a-b3ac-cbd739327d4d" (UID: "f170ecd7-a0bf-469a-b3ac-cbd739327d4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:14.348915 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:14.348839 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-workload-socket" pod "f170ecd7-a0bf-469a-b3ac-cbd739327d4d" (UID: "f170ecd7-a0bf-469a-b3ac-cbd739327d4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:14.348915 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:14.348876 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istiod-ca-cert" pod "f170ecd7-a0bf-469a-b3ac-cbd739327d4d" (UID: "f170ecd7-a0bf-469a-b3ac-cbd739327d4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:06:14.349034 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:14.348933 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istio-data" pod "f170ecd7-a0bf-469a-b3ac-cbd739327d4d" (UID: "f170ecd7-a0bf-469a-b3ac-cbd739327d4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:14.349234 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:14.349215 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:14.349334 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:14.349237 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:14.349334 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:14.349252 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:14.349334 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:14.349264 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:14.349334 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:14.349277 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:14.350963 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:14.350930 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istio-podinfo" pod "f170ecd7-a0bf-469a-b3ac-cbd739327d4d" (UID: "f170ecd7-a0bf-469a-b3ac-cbd739327d4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:06:14.351123 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:14.351102 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istio-token" pod "f170ecd7-a0bf-469a-b3ac-cbd739327d4d" (UID: "f170ecd7-a0bf-469a-b3ac-cbd739327d4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:14.351262 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:14.351244 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-kube-api-access-k7g7x" pod "f170ecd7-a0bf-469a-b3ac-cbd739327d4d" (UID: "f170ecd7-a0bf-469a-b3ac-cbd739327d4d"). InnerVolumeSpecName "kube-api-access-k7g7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:14.351322 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:14.351262 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istio-envoy" pod "f170ecd7-a0bf-469a-b3ac-cbd739327d4d" (UID: "f170ecd7-a0bf-469a-b3ac-cbd739327d4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:14.450932 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:14.450826 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:14.450932 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:14.450878 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:14.450932 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:14.450890 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:14.450932 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:14.450902 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7g7x\" (UniqueName: \"kubernetes.io/projected/f170ecd7-a0bf-469a-b3ac-cbd739327d4d-kube-api-access-k7g7x\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:14.655890 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:14.655828 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:06:15.109116 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:15.109086 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" Oct 30 18:06:15.109116 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:15.109110 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll" event={"ID":"f170ecd7-a0bf-469a-b3ac-cbd739327d4d","Type":"ContainerDied","Data":"414f5abbfd735e3d3cac69dd4e8533143c1d2286b206f1504e0f1ae3993458a3"} Oct 30 18:06:15.109632 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:15.109173 4965 scope.go:122] "RemoveContainer" containerID="4a6d4e5b44446802b627b278445a8fb338e3d490ce429967110336398727914b" Oct 30 18:06:15.162076 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:15.162045 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll"] Oct 30 18:06:15.173737 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:15.173710 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-dq6ll"] Oct 30 18:06:16.099812 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:16.099773 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f170ecd7-a0bf-469a-b3ac-cbd739327d4d" path="/var/lib/kubelet/pods/f170ecd7-a0bf-469a-b3ac-cbd739327d4d/volumes" Oct 30 18:06:17.126821 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:17.126789 4965 generic.go:356] "Generic (PLEG): container finished" podID="1a68dfa4-44fa-40d0-bf51-ecac52a252d7" containerID="6ae936b4b049bef952eb1349149bcbfa65ffe5c99b8093ef79a93d70f8be7d62" exitCode=0 Oct 30 18:06:17.127438 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:17.126879 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8d1062-595d67866c-92d2c" event={"ID":"1a68dfa4-44fa-40d0-bf51-ecac52a252d7","Type":"ContainerDied","Data":"6ae936b4b049bef952eb1349149bcbfa65ffe5c99b8093ef79a93d70f8be7d62"} Oct 30 18:06:17.265299 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:17.265230 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8d1062-595d67866c-92d2c" Oct 30 18:06:17.296311 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:17.296255 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a68dfa4-44fa-40d0-bf51-ecac52a252d7-kube-api-access-jzvgj\" (UniqueName: \"kubernetes.io/projected/1a68dfa4-44fa-40d0-bf51-ecac52a252d7-kube-api-access-jzvgj\") pod \"1a68dfa4-44fa-40d0-bf51-ecac52a252d7\" (UID: \"1a68dfa4-44fa-40d0-bf51-ecac52a252d7\") " Oct 30 18:06:17.296525 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:17.296362 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a68dfa4-44fa-40d0-bf51-ecac52a252d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a68dfa4-44fa-40d0-bf51-ecac52a252d7-plugins-config-volume\") pod \"1a68dfa4-44fa-40d0-bf51-ecac52a252d7\" (UID: \"1a68dfa4-44fa-40d0-bf51-ecac52a252d7\") " Oct 30 18:06:17.297491 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:17.297440 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a68dfa4-44fa-40d0-bf51-ecac52a252d7-plugins-config-volume" pod "1a68dfa4-44fa-40d0-bf51-ecac52a252d7" (UID: "1a68dfa4-44fa-40d0-bf51-ecac52a252d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:06:17.311889 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:17.311836 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a68dfa4-44fa-40d0-bf51-ecac52a252d7-kube-api-access-jzvgj" pod "1a68dfa4-44fa-40d0-bf51-ecac52a252d7" (UID: "1a68dfa4-44fa-40d0-bf51-ecac52a252d7"). InnerVolumeSpecName "kube-api-access-jzvgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:17.398568 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:17.398531 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzvgj\" (UniqueName: \"kubernetes.io/projected/1a68dfa4-44fa-40d0-bf51-ecac52a252d7-kube-api-access-jzvgj\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:17.398568 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:17.398568 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a68dfa4-44fa-40d0-bf51-ecac52a252d7-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:18.134922 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:18.134892 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8d1062-595d67866c-92d2c" Oct 30 18:06:18.135339 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:18.134899 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8d1062-595d67866c-92d2c" event={"ID":"1a68dfa4-44fa-40d0-bf51-ecac52a252d7","Type":"ContainerDied","Data":"ef1e98a322254af16ad8449380fb8707cadf686b3214837f5aa592f110adc0ed"} Oct 30 18:06:18.135339 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:18.135019 4965 scope.go:122] "RemoveContainer" containerID="6ae936b4b049bef952eb1349149bcbfa65ffe5c99b8093ef79a93d70f8be7d62" Oct 30 18:06:18.188519 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:18.188489 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8d1062-595d67866c-92d2c"] Oct 30 18:06:18.200518 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:18.200487 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8d1062-595d67866c-92d2c"] Oct 30 18:06:19.148504 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.148439 4965 generic.go:356] "Generic (PLEG): container finished" podID="82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e" containerID="f298491a0f8a7f978b59eca8e56c6796ee878ba49feb6f86491d7a6834ebe302" exitCode=0 Oct 30 18:06:19.148889 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.148507 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" event={"ID":"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e","Type":"ContainerDied","Data":"f298491a0f8a7f978b59eca8e56c6796ee878ba49feb6f86491d7a6834ebe302"} Oct 30 18:06:19.148889 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.148557 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" event={"ID":"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e","Type":"ContainerDied","Data":"31dd8b0c4bfd1b74a298b4ae4750ffeb653b50bc59da82ab548a9244e3c34b86"} Oct 30 18:06:19.148889 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.148577 4965 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31dd8b0c4bfd1b74a298b4ae4750ffeb653b50bc59da82ab548a9244e3c34b86" Oct 30 18:06:19.148889 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.148591 4965 scope.go:122] "RemoveContainer" containerID="56e8be9eeb4e8291cd6e6ad01953970ad164de022a156f9d86511a974b13cfae" Oct 30 18:06:19.148889 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.148642 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 18:06:19.222436 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.222400 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istiod-ca-cert\") pod \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " Oct 30 18:06:19.222618 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.222559 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-workload-socket\") pod \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " Oct 30 18:06:19.222691 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.222614 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-credential-socket\") pod \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " Oct 30 18:06:19.222691 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.222659 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-kube-api-access-422h5\" (UniqueName: \"kubernetes.io/projected/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-kube-api-access-422h5\") pod \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " Oct 30 18:06:19.222808 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.222720 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istio-envoy\") pod \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " Oct 30 18:06:19.222808 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.222761 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istio-podinfo\") pod \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " Oct 30 18:06:19.222970 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.222823 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istio-data\") pod \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " Oct 30 18:06:19.222970 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.222926 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istio-token\" (UniqueName: \"kubernetes.io/projected/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istio-token\") pod \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " Oct 30 18:06:19.222970 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.222939 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-workload-socket" pod "82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e" (UID: "82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:19.223155 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.222974 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-credential-socket" pod "82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e" (UID: "82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:19.223155 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.223038 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-workload-certs\") pod \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\" (UID: \"82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e\") " Oct 30 18:06:19.223270 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.223207 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istiod-ca-cert" pod "82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e" (UID: "82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:06:19.223462 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.223442 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istio-data" pod "82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e" (UID: "82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:19.223623 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.223604 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-workload-certs" pod "82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e" (UID: "82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:19.223783 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.223629 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:19.223923 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.223905 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:19.224025 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.223927 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:19.224025 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.223940 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:19.225637 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.225604 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istio-podinfo" pod "82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e" (UID: "82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:06:19.225807 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.225786 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istio-token" pod "82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e" (UID: "82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:19.225895 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.225821 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-kube-api-access-422h5" pod "82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e" (UID: "82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e"). InnerVolumeSpecName "kube-api-access-422h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:19.225946 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.225900 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istio-envoy" pod "82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e" (UID: "82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:19.324979 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.324946 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-422h5\" (UniqueName: \"kubernetes.io/projected/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-kube-api-access-422h5\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:19.324979 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.324975 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:19.324979 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.324987 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:19.325301 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.324996 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:19.325301 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:19.325004 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:20.099331 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:20.099295 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a68dfa4-44fa-40d0-bf51-ecac52a252d7" path="/var/lib/kubelet/pods/1a68dfa4-44fa-40d0-bf51-ecac52a252d7/volumes" Oct 30 18:06:20.156589 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:20.156564 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m" Oct 30 18:06:20.256690 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:20.256658 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m"] Oct 30 18:06:20.263080 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:20.263054 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-8d1062-openshift-default-65f489596f-mnk4m"] Oct 30 18:06:20.676333 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:20.676288 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd"] Oct 30 18:06:20.676575 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:20.676527 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd" podUID="57095c88-ac35-4dc9-b3da-4237fcf6a752" containerName="maas-api" containerID="cri-o://aa51712a2af7641a31cfc6b0bd47b1a8a5c877852a79abf0ebf86b5332a4f655" gracePeriod=30 Oct 30 18:06:20.746183 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:20.746141 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" containerID="cri-o://22c05af1832435c89516450b0cbc3a88eff16b9877d5c2cfc707f8703c65de95" gracePeriod=45 Oct 30 18:06:21.048826 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.048777 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd" Oct 30 18:06:21.133320 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.133289 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 18:06:21.150140 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.150114 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57095c88-ac35-4dc9-b3da-4237fcf6a752-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57095c88-ac35-4dc9-b3da-4237fcf6a752-openshift-service-ca\") pod \"57095c88-ac35-4dc9-b3da-4237fcf6a752\" (UID: \"57095c88-ac35-4dc9-b3da-4237fcf6a752\") " Oct 30 18:06:21.150320 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.150172 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57095c88-ac35-4dc9-b3da-4237fcf6a752-kube-api-access-n7xrn\" (UniqueName: \"kubernetes.io/projected/57095c88-ac35-4dc9-b3da-4237fcf6a752-kube-api-access-n7xrn\") pod \"57095c88-ac35-4dc9-b3da-4237fcf6a752\" (UID: \"57095c88-ac35-4dc9-b3da-4237fcf6a752\") " Oct 30 18:06:21.150320 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.150279 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57095c88-ac35-4dc9-b3da-4237fcf6a752-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57095c88-ac35-4dc9-b3da-4237fcf6a752-maas-api-tls\") pod \"57095c88-ac35-4dc9-b3da-4237fcf6a752\" (UID: \"57095c88-ac35-4dc9-b3da-4237fcf6a752\") " Oct 30 18:06:21.150444 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.150343 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57095c88-ac35-4dc9-b3da-4237fcf6a752-metrics-certs\" (UniqueName: \"kubernetes.io/secret/57095c88-ac35-4dc9-b3da-4237fcf6a752-metrics-certs\") pod \"57095c88-ac35-4dc9-b3da-4237fcf6a752\" (UID: \"57095c88-ac35-4dc9-b3da-4237fcf6a752\") " Oct 30 18:06:21.150549 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.150527 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57095c88-ac35-4dc9-b3da-4237fcf6a752-openshift-service-ca" pod "57095c88-ac35-4dc9-b3da-4237fcf6a752" (UID: "57095c88-ac35-4dc9-b3da-4237fcf6a752"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:06:21.150834 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.150811 4965 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57095c88-ac35-4dc9-b3da-4237fcf6a752-openshift-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:21.152388 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.152364 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57095c88-ac35-4dc9-b3da-4237fcf6a752-maas-api-tls" pod "57095c88-ac35-4dc9-b3da-4237fcf6a752" (UID: "57095c88-ac35-4dc9-b3da-4237fcf6a752"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:06:21.152388 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.152382 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57095c88-ac35-4dc9-b3da-4237fcf6a752-metrics-certs" pod "57095c88-ac35-4dc9-b3da-4237fcf6a752" (UID: "57095c88-ac35-4dc9-b3da-4237fcf6a752"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:06:21.152516 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.152373 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57095c88-ac35-4dc9-b3da-4237fcf6a752-kube-api-access-n7xrn" pod "57095c88-ac35-4dc9-b3da-4237fcf6a752" (UID: "57095c88-ac35-4dc9-b3da-4237fcf6a752"). InnerVolumeSpecName "kube-api-access-n7xrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:21.164605 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.164577 4965 generic.go:356] "Generic (PLEG): container finished" podID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerID="22c05af1832435c89516450b0cbc3a88eff16b9877d5c2cfc707f8703c65de95" exitCode=0 Oct 30 18:06:21.164983 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.164621 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" Oct 30 18:06:21.164983 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.164660 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" event={"ID":"46254fda-8014-4da3-9b02-beb7a1a1418c","Type":"ContainerDied","Data":"22c05af1832435c89516450b0cbc3a88eff16b9877d5c2cfc707f8703c65de95"} Oct 30 18:06:21.164983 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.164721 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" event={"ID":"46254fda-8014-4da3-9b02-beb7a1a1418c","Type":"ContainerDied","Data":"13a90a7d8c93bc6e6404f42f03ccb9288a1243208d13f3f4901b57657da9c158"} Oct 30 18:06:21.164983 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.164747 4965 scope.go:122] "RemoveContainer" containerID="22c05af1832435c89516450b0cbc3a88eff16b9877d5c2cfc707f8703c65de95" Oct 30 18:06:21.166307 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.166288 4965 generic.go:356] "Generic (PLEG): container finished" podID="57095c88-ac35-4dc9-b3da-4237fcf6a752" containerID="aa51712a2af7641a31cfc6b0bd47b1a8a5c877852a79abf0ebf86b5332a4f655" exitCode=0 Oct 30 18:06:21.166414 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.166336 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd" Oct 30 18:06:21.166414 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.166375 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd" event={"ID":"57095c88-ac35-4dc9-b3da-4237fcf6a752","Type":"ContainerDied","Data":"aa51712a2af7641a31cfc6b0bd47b1a8a5c877852a79abf0ebf86b5332a4f655"} Oct 30 18:06:21.166547 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.166421 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd" event={"ID":"57095c88-ac35-4dc9-b3da-4237fcf6a752","Type":"ContainerDied","Data":"766e1225921b1c60dd45f34c38e8a12f24c481f67a3dc1e269900d20f503a686"} Oct 30 18:06:21.177403 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.177124 4965 scope.go:122] "RemoveContainer" containerID="22c05af1832435c89516450b0cbc3a88eff16b9877d5c2cfc707f8703c65de95" Oct 30 18:06:21.177990 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:06:21.177964 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22c05af1832435c89516450b0cbc3a88eff16b9877d5c2cfc707f8703c65de95\": container with ID starting with 22c05af1832435c89516450b0cbc3a88eff16b9877d5c2cfc707f8703c65de95 not found: ID does not exist" containerID="22c05af1832435c89516450b0cbc3a88eff16b9877d5c2cfc707f8703c65de95" Oct 30 18:06:21.178085 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.177997 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22c05af1832435c89516450b0cbc3a88eff16b9877d5c2cfc707f8703c65de95"} err="failed to get container status \"22c05af1832435c89516450b0cbc3a88eff16b9877d5c2cfc707f8703c65de95\": rpc error: code = NotFound desc = could not find container \"22c05af1832435c89516450b0cbc3a88eff16b9877d5c2cfc707f8703c65de95\": container with ID starting with 22c05af1832435c89516450b0cbc3a88eff16b9877d5c2cfc707f8703c65de95 not found: ID does not exist" Oct 30 18:06:21.178085 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.178012 4965 scope.go:122] "RemoveContainer" containerID="aa51712a2af7641a31cfc6b0bd47b1a8a5c877852a79abf0ebf86b5332a4f655" Oct 30 18:06:21.206085 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.206055 4965 scope.go:122] "RemoveContainer" containerID="aa51712a2af7641a31cfc6b0bd47b1a8a5c877852a79abf0ebf86b5332a4f655" Oct 30 18:06:21.206377 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:06:21.206352 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa51712a2af7641a31cfc6b0bd47b1a8a5c877852a79abf0ebf86b5332a4f655\": container with ID starting with aa51712a2af7641a31cfc6b0bd47b1a8a5c877852a79abf0ebf86b5332a4f655 not found: ID does not exist" containerID="aa51712a2af7641a31cfc6b0bd47b1a8a5c877852a79abf0ebf86b5332a4f655" Oct 30 18:06:21.206455 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.206384 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa51712a2af7641a31cfc6b0bd47b1a8a5c877852a79abf0ebf86b5332a4f655"} err="failed to get container status \"aa51712a2af7641a31cfc6b0bd47b1a8a5c877852a79abf0ebf86b5332a4f655\": rpc error: code = NotFound desc = could not find container \"aa51712a2af7641a31cfc6b0bd47b1a8a5c877852a79abf0ebf86b5332a4f655\": container with ID starting with aa51712a2af7641a31cfc6b0bd47b1a8a5c877852a79abf0ebf86b5332a4f655 not found: ID does not exist" Oct 30 18:06:21.249134 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.249062 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd"] Oct 30 18:06:21.251542 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.251493 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46254fda-8014-4da3-9b02-beb7a1a1418c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46254fda-8014-4da3-9b02-beb7a1a1418c-model-cache\") pod \"46254fda-8014-4da3-9b02-beb7a1a1418c\" (UID: \"46254fda-8014-4da3-9b02-beb7a1a1418c\") " Oct 30 18:06:21.251730 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.251656 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46254fda-8014-4da3-9b02-beb7a1a1418c-kube-api-access-dkx7k\" (UniqueName: \"kubernetes.io/projected/46254fda-8014-4da3-9b02-beb7a1a1418c-kube-api-access-dkx7k\") pod \"46254fda-8014-4da3-9b02-beb7a1a1418c\" (UID: \"46254fda-8014-4da3-9b02-beb7a1a1418c\") " Oct 30 18:06:21.251730 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.251712 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46254fda-8014-4da3-9b02-beb7a1a1418c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/46254fda-8014-4da3-9b02-beb7a1a1418c-dshm\") pod \"46254fda-8014-4da3-9b02-beb7a1a1418c\" (UID: \"46254fda-8014-4da3-9b02-beb7a1a1418c\") " Oct 30 18:06:21.251903 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.251738 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46254fda-8014-4da3-9b02-beb7a1a1418c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46254fda-8014-4da3-9b02-beb7a1a1418c-tmp-dir\") pod \"46254fda-8014-4da3-9b02-beb7a1a1418c\" (UID: \"46254fda-8014-4da3-9b02-beb7a1a1418c\") " Oct 30 18:06:21.251903 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.251834 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46254fda-8014-4da3-9b02-beb7a1a1418c-home\" (UniqueName: \"kubernetes.io/empty-dir/46254fda-8014-4da3-9b02-beb7a1a1418c-home\") pod \"46254fda-8014-4da3-9b02-beb7a1a1418c\" (UID: \"46254fda-8014-4da3-9b02-beb7a1a1418c\") " Oct 30 18:06:21.252074 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.251916 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46254fda-8014-4da3-9b02-beb7a1a1418c-tls-certs\" (UniqueName: \"kubernetes.io/secret/46254fda-8014-4da3-9b02-beb7a1a1418c-tls-certs\") pod \"46254fda-8014-4da3-9b02-beb7a1a1418c\" (UID: \"46254fda-8014-4da3-9b02-beb7a1a1418c\") " Oct 30 18:06:21.252472 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.252408 4965 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57095c88-ac35-4dc9-b3da-4237fcf6a752-metrics-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:21.252472 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.252437 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7xrn\" (UniqueName: \"kubernetes.io/projected/57095c88-ac35-4dc9-b3da-4237fcf6a752-kube-api-access-n7xrn\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:21.252472 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.252453 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57095c88-ac35-4dc9-b3da-4237fcf6a752-maas-api-tls\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:21.252656 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.252504 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46254fda-8014-4da3-9b02-beb7a1a1418c-model-cache" pod "46254fda-8014-4da3-9b02-beb7a1a1418c" (UID: "46254fda-8014-4da3-9b02-beb7a1a1418c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:21.253056 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.253023 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46254fda-8014-4da3-9b02-beb7a1a1418c-tmp-dir" pod "46254fda-8014-4da3-9b02-beb7a1a1418c" (UID: "46254fda-8014-4da3-9b02-beb7a1a1418c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:21.253251 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.253054 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46254fda-8014-4da3-9b02-beb7a1a1418c-home" pod "46254fda-8014-4da3-9b02-beb7a1a1418c" (UID: "46254fda-8014-4da3-9b02-beb7a1a1418c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:21.255523 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.255489 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46254fda-8014-4da3-9b02-beb7a1a1418c-dshm" pod "46254fda-8014-4da3-9b02-beb7a1a1418c" (UID: "46254fda-8014-4da3-9b02-beb7a1a1418c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:21.256270 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.256245 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016f1d-8597594d96-dwhbd"] Oct 30 18:06:21.267881 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.258175 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46254fda-8014-4da3-9b02-beb7a1a1418c-tls-certs" pod "46254fda-8014-4da3-9b02-beb7a1a1418c" (UID: "46254fda-8014-4da3-9b02-beb7a1a1418c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:06:21.267881 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.258193 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46254fda-8014-4da3-9b02-beb7a1a1418c-kube-api-access-dkx7k" pod "46254fda-8014-4da3-9b02-beb7a1a1418c" (UID: "46254fda-8014-4da3-9b02-beb7a1a1418c"). InnerVolumeSpecName "kube-api-access-dkx7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:21.353679 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.353649 4965 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46254fda-8014-4da3-9b02-beb7a1a1418c-dshm\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:21.353824 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.353696 4965 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46254fda-8014-4da3-9b02-beb7a1a1418c-tmp-dir\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:21.353824 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.353707 4965 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46254fda-8014-4da3-9b02-beb7a1a1418c-home\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:21.353824 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.353715 4965 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46254fda-8014-4da3-9b02-beb7a1a1418c-tls-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:21.353824 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.353734 4965 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46254fda-8014-4da3-9b02-beb7a1a1418c-model-cache\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:21.353824 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.353744 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkx7k\" (UniqueName: \"kubernetes.io/projected/46254fda-8014-4da3-9b02-beb7a1a1418c-kube-api-access-dkx7k\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:21.511793 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.511759 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll"] Oct 30 18:06:21.523842 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:21.523816 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll"] Oct 30 18:06:22.054947 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:22.054904 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-db98b0/distinct-w5-db98b0-kserve-7d59d9bddf-ljzll" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:06:22.098830 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:22.098796 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" path="/var/lib/kubelet/pods/46254fda-8014-4da3-9b02-beb7a1a1418c/volumes" Oct 30 18:06:22.099291 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:22.099276 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57095c88-ac35-4dc9-b3da-4237fcf6a752" path="/var/lib/kubelet/pods/57095c88-ac35-4dc9-b3da-4237fcf6a752/volumes" Oct 30 18:06:22.099706 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:22.099693 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e" path="/var/lib/kubelet/pods/82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e/volumes" Oct 30 18:06:23.671780 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:23.671747 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46"] Oct 30 18:06:23.672276 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:23.672003 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46" podUID="67332f71-74ce-4486-a341-cd32c7a3a9ef" containerName="maas-api" containerID="cri-o://31b8431181707a38db23e2dbb9e3ea33af7782f11f709dad2edb8503bd5ae6d8" gracePeriod=30 Oct 30 18:06:23.939287 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:23.939222 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46" Oct 30 18:06:23.988408 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:23.985808 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67332f71-74ce-4486-a341-cd32c7a3a9ef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/67332f71-74ce-4486-a341-cd32c7a3a9ef-metrics-certs\") pod \"67332f71-74ce-4486-a341-cd32c7a3a9ef\" (UID: \"67332f71-74ce-4486-a341-cd32c7a3a9ef\") " Oct 30 18:06:23.988408 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:23.985883 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67332f71-74ce-4486-a341-cd32c7a3a9ef-kube-api-access-2lblv\" (UniqueName: \"kubernetes.io/projected/67332f71-74ce-4486-a341-cd32c7a3a9ef-kube-api-access-2lblv\") pod \"67332f71-74ce-4486-a341-cd32c7a3a9ef\" (UID: \"67332f71-74ce-4486-a341-cd32c7a3a9ef\") " Oct 30 18:06:23.988408 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:23.986009 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67332f71-74ce-4486-a341-cd32c7a3a9ef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67332f71-74ce-4486-a341-cd32c7a3a9ef-openshift-service-ca\") pod \"67332f71-74ce-4486-a341-cd32c7a3a9ef\" (UID: \"67332f71-74ce-4486-a341-cd32c7a3a9ef\") " Oct 30 18:06:23.988408 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:23.986163 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67332f71-74ce-4486-a341-cd32c7a3a9ef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67332f71-74ce-4486-a341-cd32c7a3a9ef-maas-api-tls\") pod \"67332f71-74ce-4486-a341-cd32c7a3a9ef\" (UID: \"67332f71-74ce-4486-a341-cd32c7a3a9ef\") " Oct 30 18:06:23.988408 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:23.987591 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67332f71-74ce-4486-a341-cd32c7a3a9ef-openshift-service-ca" pod "67332f71-74ce-4486-a341-cd32c7a3a9ef" (UID: "67332f71-74ce-4486-a341-cd32c7a3a9ef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:06:23.990318 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:23.989813 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67332f71-74ce-4486-a341-cd32c7a3a9ef-kube-api-access-2lblv" pod "67332f71-74ce-4486-a341-cd32c7a3a9ef" (UID: "67332f71-74ce-4486-a341-cd32c7a3a9ef"). InnerVolumeSpecName "kube-api-access-2lblv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:23.993366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:23.993342 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67332f71-74ce-4486-a341-cd32c7a3a9ef-metrics-certs" pod "67332f71-74ce-4486-a341-cd32c7a3a9ef" (UID: "67332f71-74ce-4486-a341-cd32c7a3a9ef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:06:23.993473 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:23.993381 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67332f71-74ce-4486-a341-cd32c7a3a9ef-maas-api-tls" pod "67332f71-74ce-4486-a341-cd32c7a3a9ef" (UID: "67332f71-74ce-4486-a341-cd32c7a3a9ef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:06:24.087483 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:24.087452 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67332f71-74ce-4486-a341-cd32c7a3a9ef-maas-api-tls\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:24.087483 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:24.087480 4965 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67332f71-74ce-4486-a341-cd32c7a3a9ef-metrics-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:24.087690 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:24.087491 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lblv\" (UniqueName: \"kubernetes.io/projected/67332f71-74ce-4486-a341-cd32c7a3a9ef-kube-api-access-2lblv\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:24.087690 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:24.087501 4965 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67332f71-74ce-4486-a341-cd32c7a3a9ef-openshift-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:24.194515 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:24.194423 4965 generic.go:356] "Generic (PLEG): container finished" podID="67332f71-74ce-4486-a341-cd32c7a3a9ef" containerID="31b8431181707a38db23e2dbb9e3ea33af7782f11f709dad2edb8503bd5ae6d8" exitCode=0 Oct 30 18:06:24.194694 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:24.194555 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46" event={"ID":"67332f71-74ce-4486-a341-cd32c7a3a9ef","Type":"ContainerDied","Data":"31b8431181707a38db23e2dbb9e3ea33af7782f11f709dad2edb8503bd5ae6d8"} Oct 30 18:06:24.194694 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:24.194608 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46" event={"ID":"67332f71-74ce-4486-a341-cd32c7a3a9ef","Type":"ContainerDied","Data":"81226f587be11aac89e4bc447e865eb537ba763da61e568ccd104c88e1ed1db8"} Oct 30 18:06:24.194694 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:24.194638 4965 scope.go:122] "RemoveContainer" containerID="31b8431181707a38db23e2dbb9e3ea33af7782f11f709dad2edb8503bd5ae6d8" Oct 30 18:06:24.194935 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:24.194778 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46" Oct 30 18:06:24.207629 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:24.207607 4965 scope.go:122] "RemoveContainer" containerID="31b8431181707a38db23e2dbb9e3ea33af7782f11f709dad2edb8503bd5ae6d8" Oct 30 18:06:24.207972 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:06:24.207947 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31b8431181707a38db23e2dbb9e3ea33af7782f11f709dad2edb8503bd5ae6d8\": container with ID starting with 31b8431181707a38db23e2dbb9e3ea33af7782f11f709dad2edb8503bd5ae6d8 not found: ID does not exist" containerID="31b8431181707a38db23e2dbb9e3ea33af7782f11f709dad2edb8503bd5ae6d8" Oct 30 18:06:24.208027 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:24.207980 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31b8431181707a38db23e2dbb9e3ea33af7782f11f709dad2edb8503bd5ae6d8"} err="failed to get container status \"31b8431181707a38db23e2dbb9e3ea33af7782f11f709dad2edb8503bd5ae6d8\": rpc error: code = NotFound desc = could not find container \"31b8431181707a38db23e2dbb9e3ea33af7782f11f709dad2edb8503bd5ae6d8\": container with ID starting with 31b8431181707a38db23e2dbb9e3ea33af7782f11f709dad2edb8503bd5ae6d8 not found: ID does not exist" Oct 30 18:06:24.246632 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:24.246600 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46"] Oct 30 18:06:24.252718 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:24.252694 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-455086-6d885b7d4f-4sc46"] Oct 30 18:06:24.897889 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:24.897835 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-4dcdbb-6bcbbcccc6-sbssx"] Oct 30 18:06:24.930789 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:24.930754 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-4dcdbb-77d699b46c-qm59b"] Oct 30 18:06:26.099585 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:26.099544 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67332f71-74ce-4486-a341-cd32c7a3a9ef" path="/var/lib/kubelet/pods/67332f71-74ce-4486-a341-cd32c7a3a9ef/volumes" Oct 30 18:06:26.916267 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:26.916214 4965 scope.go:122] "RemoveContainer" containerID="df7e7f18a0b17a2c3bc90a595d12875a5534f31d2350033cf1effc445ef01eae" Oct 30 18:06:26.928189 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:26.928168 4965 scope.go:122] "RemoveContainer" containerID="f298491a0f8a7f978b59eca8e56c6796ee878ba49feb6f86491d7a6834ebe302" Oct 30 18:06:26.937716 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:26.937697 4965 scope.go:122] "RemoveContainer" containerID="3bbc8ff13be2e68bfe181e89ec68284fa26f57944b4d09aed551229ce277c1f6" Oct 30 18:06:26.946587 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:26.946563 4965 scope.go:122] "RemoveContainer" containerID="bcc554a350e0719015bd25b874f26c48236a8b484aef03259a575bc0dee24e1f" Oct 30 18:06:29.898854 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:29.898811 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-4dcdbb-6bcbbcccc6-sbssx" podUID="5c0c2a4a-8305-4fc2-a440-56d6725b1ba8" containerName="payload-processing" containerID="cri-o://2588799f55a52948f9f4338ac2bf5adb30bc6e5b0a1de4c0aa80f16dd5dccfa4" gracePeriod=25 Oct 30 18:06:29.931651 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:29.931619 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-4dcdbb-77d699b46c-qm59b" podUID="8dec2e67-e439-47cc-bf7b-b315dae10d2e" containerName="payload-pre-processing" containerID="cri-o://afec01a666dc208543de81adf6c7eb40a06822ca389eeb687794293aed55a571" gracePeriod=25 Oct 30 18:06:30.179476 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:30.179444 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-4dcdbb-6bcbbcccc6-sbssx" Oct 30 18:06:30.239016 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:30.238985 4965 generic.go:356] "Generic (PLEG): container finished" podID="5c0c2a4a-8305-4fc2-a440-56d6725b1ba8" containerID="2588799f55a52948f9f4338ac2bf5adb30bc6e5b0a1de4c0aa80f16dd5dccfa4" exitCode=0 Oct 30 18:06:30.239224 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:30.239031 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-4dcdbb-6bcbbcccc6-sbssx" Oct 30 18:06:30.239224 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:30.239069 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-4dcdbb-6bcbbcccc6-sbssx" event={"ID":"5c0c2a4a-8305-4fc2-a440-56d6725b1ba8","Type":"ContainerDied","Data":"2588799f55a52948f9f4338ac2bf5adb30bc6e5b0a1de4c0aa80f16dd5dccfa4"} Oct 30 18:06:30.239224 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:30.239136 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-4dcdbb-6bcbbcccc6-sbssx" event={"ID":"5c0c2a4a-8305-4fc2-a440-56d6725b1ba8","Type":"ContainerDied","Data":"a9388e616ec6decc0e48d19da3fb2529c7659c6cedcacdb10af5f1670f4f6f13"} Oct 30 18:06:30.239224 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:30.239165 4965 scope.go:122] "RemoveContainer" containerID="2588799f55a52948f9f4338ac2bf5adb30bc6e5b0a1de4c0aa80f16dd5dccfa4" Oct 30 18:06:30.250262 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:30.250244 4965 scope.go:122] "RemoveContainer" containerID="2588799f55a52948f9f4338ac2bf5adb30bc6e5b0a1de4c0aa80f16dd5dccfa4" Oct 30 18:06:30.250598 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:06:30.250577 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2588799f55a52948f9f4338ac2bf5adb30bc6e5b0a1de4c0aa80f16dd5dccfa4\": container with ID starting with 2588799f55a52948f9f4338ac2bf5adb30bc6e5b0a1de4c0aa80f16dd5dccfa4 not found: ID does not exist" containerID="2588799f55a52948f9f4338ac2bf5adb30bc6e5b0a1de4c0aa80f16dd5dccfa4" Oct 30 18:06:30.250668 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:30.250610 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2588799f55a52948f9f4338ac2bf5adb30bc6e5b0a1de4c0aa80f16dd5dccfa4"} err="failed to get container status \"2588799f55a52948f9f4338ac2bf5adb30bc6e5b0a1de4c0aa80f16dd5dccfa4\": rpc error: code = NotFound desc = could not find container \"2588799f55a52948f9f4338ac2bf5adb30bc6e5b0a1de4c0aa80f16dd5dccfa4\": container with ID starting with 2588799f55a52948f9f4338ac2bf5adb30bc6e5b0a1de4c0aa80f16dd5dccfa4 not found: ID does not exist" Oct 30 18:06:30.312403 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:30.312372 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c0c2a4a-8305-4fc2-a440-56d6725b1ba8-kube-api-access-xv5d7\" (UniqueName: \"kubernetes.io/projected/5c0c2a4a-8305-4fc2-a440-56d6725b1ba8-kube-api-access-xv5d7\") pod \"5c0c2a4a-8305-4fc2-a440-56d6725b1ba8\" (UID: \"5c0c2a4a-8305-4fc2-a440-56d6725b1ba8\") " Oct 30 18:06:30.312576 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:30.312481 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c0c2a4a-8305-4fc2-a440-56d6725b1ba8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c0c2a4a-8305-4fc2-a440-56d6725b1ba8-plugins-config-volume\") pod \"5c0c2a4a-8305-4fc2-a440-56d6725b1ba8\" (UID: \"5c0c2a4a-8305-4fc2-a440-56d6725b1ba8\") " Oct 30 18:06:30.312832 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:30.312810 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c0c2a4a-8305-4fc2-a440-56d6725b1ba8-plugins-config-volume" pod "5c0c2a4a-8305-4fc2-a440-56d6725b1ba8" (UID: "5c0c2a4a-8305-4fc2-a440-56d6725b1ba8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:06:30.313075 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:30.313058 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c0c2a4a-8305-4fc2-a440-56d6725b1ba8-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:30.314563 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:30.314542 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c0c2a4a-8305-4fc2-a440-56d6725b1ba8-kube-api-access-xv5d7" pod "5c0c2a4a-8305-4fc2-a440-56d6725b1ba8" (UID: "5c0c2a4a-8305-4fc2-a440-56d6725b1ba8"). InnerVolumeSpecName "kube-api-access-xv5d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:30.414619 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:30.414536 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv5d7\" (UniqueName: \"kubernetes.io/projected/5c0c2a4a-8305-4fc2-a440-56d6725b1ba8-kube-api-access-xv5d7\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:30.585647 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:30.585606 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-4dcdbb-6bcbbcccc6-sbssx"] Oct 30 18:06:30.588703 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:30.588680 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-4dcdbb-6bcbbcccc6-sbssx"] Oct 30 18:06:32.085975 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.085936 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1be6b995-n4hfb"] Oct 30 18:06:32.086682 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.086639 4965 cpu_manager.go:414] "containerMap: removing container" podUID="5c0c2a4a-8305-4fc2-a440-56d6725b1ba8" containerName="payload-processing" Oct 30 18:06:32.086682 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.086660 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c0c2a4a-8305-4fc2-a440-56d6725b1ba8" containerName="payload-processing" Oct 30 18:06:32.086682 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.086673 4965 cpu_manager.go:414] "containerMap: removing container" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" Oct 30 18:06:32.086682 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.086678 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.086692 4965 cpu_manager.go:414] "containerMap: removing container" podUID="82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e" containerName="istio-proxy" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.086697 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e" containerName="istio-proxy" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.086705 4965 cpu_manager.go:414] "containerMap: removing container" podUID="b003d430-64cd-4a08-9a96-6b326a534b5e" containerName="istio-proxy" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.086711 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b003d430-64cd-4a08-9a96-6b326a534b5e" containerName="istio-proxy" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.086717 4965 cpu_manager.go:414] "containerMap: removing container" podUID="306a7985-a3c4-46d3-a640-5663ace6b620" containerName="istio-proxy" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.086721 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="306a7985-a3c4-46d3-a640-5663ace6b620" containerName="istio-proxy" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.086730 4965 cpu_manager.go:414] "containerMap: removing container" podUID="67332f71-74ce-4486-a341-cd32c7a3a9ef" containerName="maas-api" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.086734 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67332f71-74ce-4486-a341-cd32c7a3a9ef" containerName="maas-api" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.086741 4965 cpu_manager.go:414] "containerMap: removing container" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.086745 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.086764 4965 cpu_manager.go:414] "containerMap: removing container" podUID="84fbd1da-e017-490a-8db0-1c9555ceb3b4" containerName="payload-pre-processing" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.086769 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84fbd1da-e017-490a-8db0-1c9555ceb3b4" containerName="payload-pre-processing" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.086777 4965 cpu_manager.go:414] "containerMap: removing container" podUID="1a68dfa4-44fa-40d0-bf51-ecac52a252d7" containerName="payload-pre-processing" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.086782 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a68dfa4-44fa-40d0-bf51-ecac52a252d7" containerName="payload-pre-processing" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.086791 4965 cpu_manager.go:414] "containerMap: removing container" podUID="17d99e83-45e9-446a-9525-ee96fe9ff19a" containerName="payload-processing" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.086795 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17d99e83-45e9-446a-9525-ee96fe9ff19a" containerName="payload-processing" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.086810 4965 cpu_manager.go:414] "containerMap: removing container" podUID="f170ecd7-a0bf-469a-b3ac-cbd739327d4d" containerName="istio-proxy" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.086814 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f170ecd7-a0bf-469a-b3ac-cbd739327d4d" containerName="istio-proxy" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.086819 4965 cpu_manager.go:414] "containerMap: removing container" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.086824 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.086832 4965 cpu_manager.go:414] "containerMap: removing container" podUID="57095c88-ac35-4dc9-b3da-4237fcf6a752" containerName="maas-api" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.086836 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57095c88-ac35-4dc9-b3da-4237fcf6a752" containerName="maas-api" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.086845 4965 cpu_manager.go:414] "containerMap: removing container" podUID="dfcedd36-934a-4257-b1e9-db368ed2bcd5" containerName="payload-pre-processing" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.086850 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfcedd36-934a-4257-b1e9-db368ed2bcd5" containerName="payload-pre-processing" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.086858 4965 cpu_manager.go:414] "containerMap: removing container" podUID="6706cbd9-edfa-4b4d-ae4a-44a037e61efc" containerName="payload-processing" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.086880 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6706cbd9-edfa-4b4d-ae4a-44a037e61efc" containerName="payload-processing" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.086996 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="f170ecd7-a0bf-469a-b3ac-cbd739327d4d" containerName="istio-proxy" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.087010 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="46254fda-8014-4da3-9b02-beb7a1a1418c" containerName="main" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.087018 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="b003d430-64cd-4a08-9a96-6b326a534b5e" containerName="istio-proxy" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.087026 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="57095c88-ac35-4dc9-b3da-4237fcf6a752" containerName="maas-api" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.087037 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfcedd36-934a-4257-b1e9-db368ed2bcd5" containerName="payload-pre-processing" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.087048 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="6706cbd9-edfa-4b4d-ae4a-44a037e61efc" containerName="payload-processing" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.087059 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c0c2a4a-8305-4fc2-a440-56d6725b1ba8" containerName="payload-processing" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.087067 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf9a4d4d-8cfa-42c8-a249-d2e1fa7deba4" containerName="main" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.087076 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e" containerName="istio-proxy" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.087084 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="b003d430-64cd-4a08-9a96-6b326a534b5e" containerName="istio-proxy" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.087093 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="306a7985-a3c4-46d3-a640-5663ace6b620" containerName="istio-proxy" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.087103 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="67332f71-74ce-4486-a341-cd32c7a3a9ef" containerName="maas-api" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.087112 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="38c90d17-2efa-4b7c-b10f-06a3a8b2a13d" containerName="main" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.087125 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="84fbd1da-e017-490a-8db0-1c9555ceb3b4" containerName="payload-pre-processing" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.087136 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a68dfa4-44fa-40d0-bf51-ecac52a252d7" containerName="payload-pre-processing" Oct 30 18:06:32.087366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.087144 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="17d99e83-45e9-446a-9525-ee96fe9ff19a" containerName="payload-processing" Oct 30 18:06:32.094457 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.094430 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1be6b995-n4hfb"] Oct 30 18:06:32.094581 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.094506 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be6b995-n4hfb" Oct 30 18:06:32.107262 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.107230 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c0c2a4a-8305-4fc2-a440-56d6725b1ba8" path="/var/lib/kubelet/pods/5c0c2a4a-8305-4fc2-a440-56d6725b1ba8/volumes" Oct 30 18:06:32.109105 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.109080 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:06:32.121064 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.121008 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\"" Oct 30 18:06:32.150331 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.150298 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsm4x\" (UniqueName: \"kubernetes.io/projected/0b751169-3514-4c01-9b26-986228c48aab-kube-api-access-hsm4x\") pod \"maas-api-revoke-sub-1be6b995-n4hfb\" (UID: \"0b751169-3514-4c01-9b26-986228c48aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be6b995-n4hfb" Oct 30 18:06:32.150507 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.150428 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b751169-3514-4c01-9b26-986228c48aab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1be6b995-n4hfb\" (UID: \"0b751169-3514-4c01-9b26-986228c48aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be6b995-n4hfb" Oct 30 18:06:32.150576 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.150549 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b751169-3514-4c01-9b26-986228c48aab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1be6b995-n4hfb\" (UID: \"0b751169-3514-4c01-9b26-986228c48aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be6b995-n4hfb" Oct 30 18:06:32.251728 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.251697 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b751169-3514-4c01-9b26-986228c48aab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1be6b995-n4hfb\" (UID: \"0b751169-3514-4c01-9b26-986228c48aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be6b995-n4hfb" Oct 30 18:06:32.251935 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.251750 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b751169-3514-4c01-9b26-986228c48aab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1be6b995-n4hfb\" (UID: \"0b751169-3514-4c01-9b26-986228c48aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be6b995-n4hfb" Oct 30 18:06:32.251935 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.251795 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsm4x\" (UniqueName: \"kubernetes.io/projected/0b751169-3514-4c01-9b26-986228c48aab-kube-api-access-hsm4x\") pod \"maas-api-revoke-sub-1be6b995-n4hfb\" (UID: \"0b751169-3514-4c01-9b26-986228c48aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be6b995-n4hfb" Oct 30 18:06:32.252528 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.252507 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b751169-3514-4c01-9b26-986228c48aab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1be6b995-n4hfb\" (UID: \"0b751169-3514-4c01-9b26-986228c48aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be6b995-n4hfb" Oct 30 18:06:32.254173 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.254148 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b751169-3514-4c01-9b26-986228c48aab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1be6b995-n4hfb\" (UID: \"0b751169-3514-4c01-9b26-986228c48aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be6b995-n4hfb" Oct 30 18:06:32.277319 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.277290 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsm4x\" (UniqueName: \"kubernetes.io/projected/0b751169-3514-4c01-9b26-986228c48aab-kube-api-access-hsm4x\") pod \"maas-api-revoke-sub-1be6b995-n4hfb\" (UID: \"0b751169-3514-4c01-9b26-986228c48aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be6b995-n4hfb" Oct 30 18:06:32.413132 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.413045 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be6b995-n4hfb" Oct 30 18:06:32.715151 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:32.715126 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1be6b995-n4hfb"] Oct 30 18:06:32.716981 ip-10-0-139-222 kubenswrapper[4965]: W1030 18:06:32.716949 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b751169_3514_4c01_9b26_986228c48aab.slice/crio-17c4dfb5dc0707d6d18ea24df1ffbacbecd2c08178479ccd9581095922d1a23b WatchSource:0}: Error finding container 17c4dfb5dc0707d6d18ea24df1ffbacbecd2c08178479ccd9581095922d1a23b: Status 404 returned error can't find the container with id 17c4dfb5dc0707d6d18ea24df1ffbacbecd2c08178479ccd9581095922d1a23b Oct 30 18:06:33.273684 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:33.273604 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be6b995-n4hfb" event={"ID":"0b751169-3514-4c01-9b26-986228c48aab","Type":"ContainerStarted","Data":"6a2b87a90e8346c823b44d6d5a0b80de37f87fa790f9874c4ae47e583b173125"} Oct 30 18:06:33.273684 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:33.273656 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be6b995-n4hfb" event={"ID":"0b751169-3514-4c01-9b26-986228c48aab","Type":"ContainerStarted","Data":"17c4dfb5dc0707d6d18ea24df1ffbacbecd2c08178479ccd9581095922d1a23b"} Oct 30 18:06:33.323590 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:33.323528 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be6b995-n4hfb" podStartSLOduration=1.323509531 podStartE2EDuration="1.323509531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:33.321961176 +0000 UTC m=+3251.374061120" watchObservedRunningTime="2026-10-30 18:06:33.323509531 +0000 UTC m=+3251.375609462" Oct 30 18:06:34.280643 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:34.280607 4965 generic.go:356] "Generic (PLEG): container finished" podID="0b751169-3514-4c01-9b26-986228c48aab" containerID="6a2b87a90e8346c823b44d6d5a0b80de37f87fa790f9874c4ae47e583b173125" exitCode=0 Oct 30 18:06:34.281331 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:34.280693 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be6b995-n4hfb" event={"ID":"0b751169-3514-4c01-9b26-986228c48aab","Type":"ContainerDied","Data":"6a2b87a90e8346c823b44d6d5a0b80de37f87fa790f9874c4ae47e583b173125"} Oct 30 18:06:34.907616 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:34.907572 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-4dcdbb-77d699b46c-qm59b" podUID="8dec2e67-e439-47cc-bf7b-b315dae10d2e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:06:35.438087 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:35.438064 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be6b995-n4hfb" Oct 30 18:06:35.489596 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:35.489558 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b751169-3514-4c01-9b26-986228c48aab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b751169-3514-4c01-9b26-986228c48aab-maas-api-openshift-ca\") pod \"0b751169-3514-4c01-9b26-986228c48aab\" (UID: \"0b751169-3514-4c01-9b26-986228c48aab\") " Oct 30 18:06:35.489804 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:35.489655 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b751169-3514-4c01-9b26-986228c48aab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b751169-3514-4c01-9b26-986228c48aab-maas-api-xks-ca\") pod \"0b751169-3514-4c01-9b26-986228c48aab\" (UID: \"0b751169-3514-4c01-9b26-986228c48aab\") " Oct 30 18:06:35.489804 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:35.489724 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b751169-3514-4c01-9b26-986228c48aab-kube-api-access-hsm4x\" (UniqueName: \"kubernetes.io/projected/0b751169-3514-4c01-9b26-986228c48aab-kube-api-access-hsm4x\") pod \"0b751169-3514-4c01-9b26-986228c48aab\" (UID: \"0b751169-3514-4c01-9b26-986228c48aab\") " Oct 30 18:06:35.494112 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:35.494083 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b751169-3514-4c01-9b26-986228c48aab-kube-api-access-hsm4x" pod "0b751169-3514-4c01-9b26-986228c48aab" (UID: "0b751169-3514-4c01-9b26-986228c48aab"). InnerVolumeSpecName "kube-api-access-hsm4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:35.496193 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:35.496160 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b751169-3514-4c01-9b26-986228c48aab-maas-api-openshift-ca" pod "0b751169-3514-4c01-9b26-986228c48aab" (UID: "0b751169-3514-4c01-9b26-986228c48aab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:06:35.498606 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:35.498572 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b751169-3514-4c01-9b26-986228c48aab-maas-api-xks-ca" pod "0b751169-3514-4c01-9b26-986228c48aab" (UID: "0b751169-3514-4c01-9b26-986228c48aab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:06:35.592181 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:35.592149 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b751169-3514-4c01-9b26-986228c48aab-maas-api-openshift-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:35.592181 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:35.592178 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b751169-3514-4c01-9b26-986228c48aab-maas-api-xks-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:35.592387 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:35.592189 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsm4x\" (UniqueName: \"kubernetes.io/projected/0b751169-3514-4c01-9b26-986228c48aab-kube-api-access-hsm4x\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:35.996779 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:06:35.996696 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:06:36.295774 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:36.295683 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be6b995-n4hfb" event={"ID":"0b751169-3514-4c01-9b26-986228c48aab","Type":"ContainerDied","Data":"17c4dfb5dc0707d6d18ea24df1ffbacbecd2c08178479ccd9581095922d1a23b"} Oct 30 18:06:36.295774 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:36.295706 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be6b995-n4hfb" Oct 30 18:06:36.295774 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:36.295736 4965 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17c4dfb5dc0707d6d18ea24df1ffbacbecd2c08178479ccd9581095922d1a23b" Oct 30 18:06:39.575119 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:39.575084 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-506124-59bc6cd6c4-rp7pn"] Oct 30 18:06:42.197872 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:42.197827 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:06:44.575954 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:44.575911 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-506124-59bc6cd6c4-rp7pn" podUID="6540385c-e0cf-41cd-8e45-123e600e2038" containerName="payload-processing" containerID="cri-o://4db60a6605f95d02333f5748f3e1dbb409bd4cfd1cea30df1069105acaff950f" gracePeriod=25 Oct 30 18:06:44.906909 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:44.906807 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-4dcdbb-77d699b46c-qm59b" podUID="8dec2e67-e439-47cc-bf7b-b315dae10d2e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:06:45.041522 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:45.041482 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-506124-59bc6cd6c4-rp7pn" podUID="6540385c-e0cf-41cd-8e45-123e600e2038" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:06:45.160705 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:45.160607 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm"] Oct 30 18:06:50.193990 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:50.193925 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-4dcdbb-77d699b46c-qm59b" Oct 30 18:06:50.312544 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:50.312495 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dec2e67-e439-47cc-bf7b-b315dae10d2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dec2e67-e439-47cc-bf7b-b315dae10d2e-plugins-config-volume\") pod \"8dec2e67-e439-47cc-bf7b-b315dae10d2e\" (UID: \"8dec2e67-e439-47cc-bf7b-b315dae10d2e\") " Oct 30 18:06:50.313583 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:50.313555 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dec2e67-e439-47cc-bf7b-b315dae10d2e-kube-api-access-srhj6\" (UniqueName: \"kubernetes.io/projected/8dec2e67-e439-47cc-bf7b-b315dae10d2e-kube-api-access-srhj6\") pod \"8dec2e67-e439-47cc-bf7b-b315dae10d2e\" (UID: \"8dec2e67-e439-47cc-bf7b-b315dae10d2e\") " Oct 30 18:06:50.314834 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:50.313281 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dec2e67-e439-47cc-bf7b-b315dae10d2e-plugins-config-volume" pod "8dec2e67-e439-47cc-bf7b-b315dae10d2e" (UID: "8dec2e67-e439-47cc-bf7b-b315dae10d2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:06:50.318447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:50.318418 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dec2e67-e439-47cc-bf7b-b315dae10d2e-kube-api-access-srhj6" pod "8dec2e67-e439-47cc-bf7b-b315dae10d2e" (UID: "8dec2e67-e439-47cc-bf7b-b315dae10d2e"). InnerVolumeSpecName "kube-api-access-srhj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:50.416645 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:50.416608 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srhj6\" (UniqueName: \"kubernetes.io/projected/8dec2e67-e439-47cc-bf7b-b315dae10d2e-kube-api-access-srhj6\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:50.416645 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:50.416640 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dec2e67-e439-47cc-bf7b-b315dae10d2e-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:50.431699 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:50.431667 4965 generic.go:356] "Generic (PLEG): container finished" podID="8dec2e67-e439-47cc-bf7b-b315dae10d2e" containerID="afec01a666dc208543de81adf6c7eb40a06822ca389eeb687794293aed55a571" exitCode=0 Oct 30 18:06:50.431927 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:50.431713 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-4dcdbb-77d699b46c-qm59b" Oct 30 18:06:50.431927 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:50.431732 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-4dcdbb-77d699b46c-qm59b" event={"ID":"8dec2e67-e439-47cc-bf7b-b315dae10d2e","Type":"ContainerDied","Data":"afec01a666dc208543de81adf6c7eb40a06822ca389eeb687794293aed55a571"} Oct 30 18:06:50.431927 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:50.431786 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-4dcdbb-77d699b46c-qm59b" event={"ID":"8dec2e67-e439-47cc-bf7b-b315dae10d2e","Type":"ContainerDied","Data":"f831da814c0282e4eb839e6cc4e2b9b374269c2fa29f784522f0a1bf5e2cdf45"} Oct 30 18:06:50.431927 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:50.431815 4965 scope.go:122] "RemoveContainer" containerID="afec01a666dc208543de81adf6c7eb40a06822ca389eeb687794293aed55a571" Oct 30 18:06:50.443020 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:50.443000 4965 scope.go:122] "RemoveContainer" containerID="afec01a666dc208543de81adf6c7eb40a06822ca389eeb687794293aed55a571" Oct 30 18:06:50.443294 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:06:50.443275 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afec01a666dc208543de81adf6c7eb40a06822ca389eeb687794293aed55a571\": container with ID starting with afec01a666dc208543de81adf6c7eb40a06822ca389eeb687794293aed55a571 not found: ID does not exist" containerID="afec01a666dc208543de81adf6c7eb40a06822ca389eeb687794293aed55a571" Oct 30 18:06:50.443335 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:50.443303 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afec01a666dc208543de81adf6c7eb40a06822ca389eeb687794293aed55a571"} err="failed to get container status \"afec01a666dc208543de81adf6c7eb40a06822ca389eeb687794293aed55a571\": rpc error: code = NotFound desc = could not find container \"afec01a666dc208543de81adf6c7eb40a06822ca389eeb687794293aed55a571\": container with ID starting with afec01a666dc208543de81adf6c7eb40a06822ca389eeb687794293aed55a571 not found: ID does not exist" Oct 30 18:06:50.483017 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:50.482946 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-4dcdbb-77d699b46c-qm59b"] Oct 30 18:06:50.488428 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:50.488404 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-4dcdbb-77d699b46c-qm59b"] Oct 30 18:06:52.105784 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:52.105751 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dec2e67-e439-47cc-bf7b-b315dae10d2e" path="/var/lib/kubelet/pods/8dec2e67-e439-47cc-bf7b-b315dae10d2e/volumes" Oct 30 18:06:53.254696 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:53.254651 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w"] Oct 30 18:06:53.255246 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:53.254882 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w" podUID="4477dfb1-111b-48c7-86ec-2a048339b95c" containerName="maas-api" containerID="cri-o://8e444329a7a1fdd59aa81889bdad2734899331284ec8cfd24365988b03bb1742" gracePeriod=30 Oct 30 18:06:53.457226 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:53.457186 4965 generic.go:356] "Generic (PLEG): container finished" podID="4477dfb1-111b-48c7-86ec-2a048339b95c" containerID="8e444329a7a1fdd59aa81889bdad2734899331284ec8cfd24365988b03bb1742" exitCode=0 Oct 30 18:06:53.457407 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:53.457305 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w" event={"ID":"4477dfb1-111b-48c7-86ec-2a048339b95c","Type":"ContainerDied","Data":"8e444329a7a1fdd59aa81889bdad2734899331284ec8cfd24365988b03bb1742"} Oct 30 18:06:53.576398 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:53.576368 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w" Oct 30 18:06:53.681327 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:53.681291 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4477dfb1-111b-48c7-86ec-2a048339b95c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4477dfb1-111b-48c7-86ec-2a048339b95c-maas-api-tls\") pod \"4477dfb1-111b-48c7-86ec-2a048339b95c\" (UID: \"4477dfb1-111b-48c7-86ec-2a048339b95c\") " Oct 30 18:06:53.681543 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:53.681362 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4477dfb1-111b-48c7-86ec-2a048339b95c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4477dfb1-111b-48c7-86ec-2a048339b95c-openshift-service-ca\") pod \"4477dfb1-111b-48c7-86ec-2a048339b95c\" (UID: \"4477dfb1-111b-48c7-86ec-2a048339b95c\") " Oct 30 18:06:53.681543 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:53.681396 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4477dfb1-111b-48c7-86ec-2a048339b95c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4477dfb1-111b-48c7-86ec-2a048339b95c-metrics-certs\") pod \"4477dfb1-111b-48c7-86ec-2a048339b95c\" (UID: \"4477dfb1-111b-48c7-86ec-2a048339b95c\") " Oct 30 18:06:53.681543 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:53.681459 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4477dfb1-111b-48c7-86ec-2a048339b95c-kube-api-access-zg7cb\" (UniqueName: \"kubernetes.io/projected/4477dfb1-111b-48c7-86ec-2a048339b95c-kube-api-access-zg7cb\") pod \"4477dfb1-111b-48c7-86ec-2a048339b95c\" (UID: \"4477dfb1-111b-48c7-86ec-2a048339b95c\") " Oct 30 18:06:53.682501 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:53.681782 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4477dfb1-111b-48c7-86ec-2a048339b95c-openshift-service-ca" pod "4477dfb1-111b-48c7-86ec-2a048339b95c" (UID: "4477dfb1-111b-48c7-86ec-2a048339b95c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:06:53.682501 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:53.682063 4965 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4477dfb1-111b-48c7-86ec-2a048339b95c-openshift-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:53.695278 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:53.695219 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4477dfb1-111b-48c7-86ec-2a048339b95c-kube-api-access-zg7cb" pod "4477dfb1-111b-48c7-86ec-2a048339b95c" (UID: "4477dfb1-111b-48c7-86ec-2a048339b95c"). InnerVolumeSpecName "kube-api-access-zg7cb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:53.696506 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:53.696473 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4477dfb1-111b-48c7-86ec-2a048339b95c-maas-api-tls" pod "4477dfb1-111b-48c7-86ec-2a048339b95c" (UID: "4477dfb1-111b-48c7-86ec-2a048339b95c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:06:53.703088 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:53.703060 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4477dfb1-111b-48c7-86ec-2a048339b95c-metrics-certs" pod "4477dfb1-111b-48c7-86ec-2a048339b95c" (UID: "4477dfb1-111b-48c7-86ec-2a048339b95c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:06:53.783708 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:53.783671 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg7cb\" (UniqueName: \"kubernetes.io/projected/4477dfb1-111b-48c7-86ec-2a048339b95c-kube-api-access-zg7cb\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:53.783708 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:53.783705 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4477dfb1-111b-48c7-86ec-2a048339b95c-maas-api-tls\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:53.783708 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:53.783717 4965 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4477dfb1-111b-48c7-86ec-2a048339b95c-metrics-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:06:54.464609 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:54.464510 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w" event={"ID":"4477dfb1-111b-48c7-86ec-2a048339b95c","Type":"ContainerDied","Data":"553901b8c2de5cc03e36a81ec2bfa52d9df7ccb65f23abbc2c14eb721d223b6b"} Oct 30 18:06:54.464609 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:54.464552 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w" Oct 30 18:06:54.464609 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:54.464576 4965 scope.go:122] "RemoveContainer" containerID="8e444329a7a1fdd59aa81889bdad2734899331284ec8cfd24365988b03bb1742" Oct 30 18:06:54.517191 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:54.517159 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w"] Oct 30 18:06:54.529142 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:54.529113 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-db98b0-7bf54b4848-49z2w"] Oct 30 18:06:55.041891 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:55.041823 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-506124-59bc6cd6c4-rp7pn" podUID="6540385c-e0cf-41cd-8e45-123e600e2038" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:06:56.098538 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:06:56.098504 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4477dfb1-111b-48c7-86ec-2a048339b95c" path="/var/lib/kubelet/pods/4477dfb1-111b-48c7-86ec-2a048339b95c/volumes" Oct 30 18:07:00.181662 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:00.181605 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" containerID="cri-o://ae510ad22726505e73796748d152e754f92b6ff802f361288fdc07b7b9eaefb8" gracePeriod=45 Oct 30 18:07:00.453562 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:00.453537 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 18:07:00.456631 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:00.456613 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f715fb84-3304-4d78-a4e6-f96c5b258e82-tls-certs\" (UniqueName: \"kubernetes.io/secret/f715fb84-3304-4d78-a4e6-f96c5b258e82-tls-certs\") pod \"f715fb84-3304-4d78-a4e6-f96c5b258e82\" (UID: \"f715fb84-3304-4d78-a4e6-f96c5b258e82\") " Oct 30 18:07:00.456724 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:00.456642 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f715fb84-3304-4d78-a4e6-f96c5b258e82-home\" (UniqueName: \"kubernetes.io/empty-dir/f715fb84-3304-4d78-a4e6-f96c5b258e82-home\") pod \"f715fb84-3304-4d78-a4e6-f96c5b258e82\" (UID: \"f715fb84-3304-4d78-a4e6-f96c5b258e82\") " Oct 30 18:07:00.456793 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:00.456772 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f715fb84-3304-4d78-a4e6-f96c5b258e82-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f715fb84-3304-4d78-a4e6-f96c5b258e82-dshm\") pod \"f715fb84-3304-4d78-a4e6-f96c5b258e82\" (UID: \"f715fb84-3304-4d78-a4e6-f96c5b258e82\") " Oct 30 18:07:00.456873 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:00.456819 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f715fb84-3304-4d78-a4e6-f96c5b258e82-kube-api-access-kv4rh\" (UniqueName: \"kubernetes.io/projected/f715fb84-3304-4d78-a4e6-f96c5b258e82-kube-api-access-kv4rh\") pod \"f715fb84-3304-4d78-a4e6-f96c5b258e82\" (UID: \"f715fb84-3304-4d78-a4e6-f96c5b258e82\") " Oct 30 18:07:00.456939 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:00.456881 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f715fb84-3304-4d78-a4e6-f96c5b258e82-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f715fb84-3304-4d78-a4e6-f96c5b258e82-model-cache\") pod \"f715fb84-3304-4d78-a4e6-f96c5b258e82\" (UID: \"f715fb84-3304-4d78-a4e6-f96c5b258e82\") " Oct 30 18:07:00.456939 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:00.456917 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f715fb84-3304-4d78-a4e6-f96c5b258e82-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f715fb84-3304-4d78-a4e6-f96c5b258e82-tmp-dir\") pod \"f715fb84-3304-4d78-a4e6-f96c5b258e82\" (UID: \"f715fb84-3304-4d78-a4e6-f96c5b258e82\") " Oct 30 18:07:00.456939 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:00.456922 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f715fb84-3304-4d78-a4e6-f96c5b258e82-home" pod "f715fb84-3304-4d78-a4e6-f96c5b258e82" (UID: "f715fb84-3304-4d78-a4e6-f96c5b258e82"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:07:00.457154 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:00.457132 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f715fb84-3304-4d78-a4e6-f96c5b258e82-model-cache" pod "f715fb84-3304-4d78-a4e6-f96c5b258e82" (UID: "f715fb84-3304-4d78-a4e6-f96c5b258e82"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:07:00.457235 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:00.457202 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f715fb84-3304-4d78-a4e6-f96c5b258e82-tmp-dir" pod "f715fb84-3304-4d78-a4e6-f96c5b258e82" (UID: "f715fb84-3304-4d78-a4e6-f96c5b258e82"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:07:00.457443 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:00.457422 4965 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f715fb84-3304-4d78-a4e6-f96c5b258e82-home\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:00.457555 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:00.457445 4965 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f715fb84-3304-4d78-a4e6-f96c5b258e82-model-cache\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:00.457555 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:00.457459 4965 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f715fb84-3304-4d78-a4e6-f96c5b258e82-tmp-dir\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:00.458975 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:00.458954 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f715fb84-3304-4d78-a4e6-f96c5b258e82-dshm" pod "f715fb84-3304-4d78-a4e6-f96c5b258e82" (UID: "f715fb84-3304-4d78-a4e6-f96c5b258e82"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:07:00.458975 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:00.458960 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f715fb84-3304-4d78-a4e6-f96c5b258e82-tls-certs" pod "f715fb84-3304-4d78-a4e6-f96c5b258e82" (UID: "f715fb84-3304-4d78-a4e6-f96c5b258e82"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:07:00.459108 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:00.459092 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f715fb84-3304-4d78-a4e6-f96c5b258e82-kube-api-access-kv4rh" pod "f715fb84-3304-4d78-a4e6-f96c5b258e82" (UID: "f715fb84-3304-4d78-a4e6-f96c5b258e82"). InnerVolumeSpecName "kube-api-access-kv4rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:07:00.502543 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:00.502509 4965 generic.go:356] "Generic (PLEG): container finished" podID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerID="ae510ad22726505e73796748d152e754f92b6ff802f361288fdc07b7b9eaefb8" exitCode=0 Oct 30 18:07:00.502904 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:00.502557 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" Oct 30 18:07:00.503556 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:00.502575 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" event={"ID":"f715fb84-3304-4d78-a4e6-f96c5b258e82","Type":"ContainerDied","Data":"ae510ad22726505e73796748d152e754f92b6ff802f361288fdc07b7b9eaefb8"} Oct 30 18:07:00.503698 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:00.503584 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm" event={"ID":"f715fb84-3304-4d78-a4e6-f96c5b258e82","Type":"ContainerDied","Data":"895175ca4e7f6f4f2085df16437ae1d613240b35bb80f339db828bb6768b9391"} Oct 30 18:07:00.503698 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:00.503613 4965 scope.go:122] "RemoveContainer" containerID="ae510ad22726505e73796748d152e754f92b6ff802f361288fdc07b7b9eaefb8" Oct 30 18:07:00.516995 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:00.516954 4965 scope.go:122] "RemoveContainer" containerID="ae510ad22726505e73796748d152e754f92b6ff802f361288fdc07b7b9eaefb8" Oct 30 18:07:00.517311 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:07:00.517289 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae510ad22726505e73796748d152e754f92b6ff802f361288fdc07b7b9eaefb8\": container with ID starting with ae510ad22726505e73796748d152e754f92b6ff802f361288fdc07b7b9eaefb8 not found: ID does not exist" containerID="ae510ad22726505e73796748d152e754f92b6ff802f361288fdc07b7b9eaefb8" Oct 30 18:07:00.517387 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:00.517318 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae510ad22726505e73796748d152e754f92b6ff802f361288fdc07b7b9eaefb8"} err="failed to get container status \"ae510ad22726505e73796748d152e754f92b6ff802f361288fdc07b7b9eaefb8\": rpc error: code = NotFound desc = could not find container \"ae510ad22726505e73796748d152e754f92b6ff802f361288fdc07b7b9eaefb8\": container with ID starting with ae510ad22726505e73796748d152e754f92b6ff802f361288fdc07b7b9eaefb8 not found: ID does not exist" Oct 30 18:07:00.559117 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:00.559044 4965 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f715fb84-3304-4d78-a4e6-f96c5b258e82-tls-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:00.559117 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:00.559078 4965 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f715fb84-3304-4d78-a4e6-f96c5b258e82-dshm\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:00.559117 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:00.559092 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv4rh\" (UniqueName: \"kubernetes.io/projected/f715fb84-3304-4d78-a4e6-f96c5b258e82-kube-api-access-kv4rh\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:00.559566 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:00.559542 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm"] Oct 30 18:07:00.563787 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:00.563757 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-8d946d/simulator-w6-8d946d-kserve-5d79f6bb9b-s57bm"] Oct 30 18:07:01.113008 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:01.109264 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5"] Oct 30 18:07:01.113008 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:01.109506 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" podUID="d8f96554-6aec-4052-8219-307fb11b6404" containerName="istio-proxy" containerID="cri-o://9c65aa992d826ddf4bd112dc532b7cded0a020ecf963add7b2cb0d8d69d2ce3d" gracePeriod=30 Oct 30 18:07:01.596926 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:01.596883 4965 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:07:01.597347 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:01.596957 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" podUID="d8f96554-6aec-4052-8219-307fb11b6404" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:07:02.098049 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:02.098016 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" path="/var/lib/kubelet/pods/f715fb84-3304-4d78-a4e6-f96c5b258e82/volumes" Oct 30 18:07:03.554378 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:03.554295 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d439241-mf8d2"] Oct 30 18:07:03.554893 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:03.554846 4965 cpu_manager.go:414] "containerMap: removing container" podUID="8dec2e67-e439-47cc-bf7b-b315dae10d2e" containerName="payload-pre-processing" Oct 30 18:07:03.554893 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:03.554874 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dec2e67-e439-47cc-bf7b-b315dae10d2e" containerName="payload-pre-processing" Oct 30 18:07:03.554993 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:03.554901 4965 cpu_manager.go:414] "containerMap: removing container" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" Oct 30 18:07:03.554993 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:03.554907 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" Oct 30 18:07:03.554993 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:03.554915 4965 cpu_manager.go:414] "containerMap: removing container" podUID="4477dfb1-111b-48c7-86ec-2a048339b95c" containerName="maas-api" Oct 30 18:07:03.554993 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:03.554920 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4477dfb1-111b-48c7-86ec-2a048339b95c" containerName="maas-api" Oct 30 18:07:03.554993 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:03.554927 4965 cpu_manager.go:414] "containerMap: removing container" podUID="82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e" containerName="istio-proxy" Oct 30 18:07:03.554993 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:03.554931 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e" containerName="istio-proxy" Oct 30 18:07:03.554993 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:03.554938 4965 cpu_manager.go:414] "containerMap: removing container" podUID="b003d430-64cd-4a08-9a96-6b326a534b5e" containerName="istio-proxy" Oct 30 18:07:03.554993 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:03.554942 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b003d430-64cd-4a08-9a96-6b326a534b5e" containerName="istio-proxy" Oct 30 18:07:03.554993 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:03.554963 4965 cpu_manager.go:414] "containerMap: removing container" podUID="0b751169-3514-4c01-9b26-986228c48aab" containerName="revoke-keys" Oct 30 18:07:03.554993 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:03.554967 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b751169-3514-4c01-9b26-986228c48aab" containerName="revoke-keys" Oct 30 18:07:03.555320 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:03.555065 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dec2e67-e439-47cc-bf7b-b315dae10d2e" containerName="payload-pre-processing" Oct 30 18:07:03.555320 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:03.555076 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="f715fb84-3304-4d78-a4e6-f96c5b258e82" containerName="main" Oct 30 18:07:03.555320 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:03.555082 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="4477dfb1-111b-48c7-86ec-2a048339b95c" containerName="maas-api" Oct 30 18:07:03.555320 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:03.555087 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="82a26e93-a3d6-4ea9-ae39-edda4cd2fc0e" containerName="istio-proxy" Oct 30 18:07:03.555320 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:03.555096 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b751169-3514-4c01-9b26-986228c48aab" containerName="revoke-keys" Oct 30 18:07:03.559823 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:03.559770 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d439241-mf8d2" Oct 30 18:07:03.579791 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:03.579646 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d439241-mf8d2"] Oct 30 18:07:03.598298 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:03.598263 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c390c71-420f-489f-9a84-0ac10e6a74ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d439241-mf8d2\" (UID: \"8c390c71-420f-489f-9a84-0ac10e6a74ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d439241-mf8d2" Oct 30 18:07:03.598467 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:03.598406 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c390c71-420f-489f-9a84-0ac10e6a74ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d439241-mf8d2\" (UID: \"8c390c71-420f-489f-9a84-0ac10e6a74ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d439241-mf8d2" Oct 30 18:07:03.598544 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:03.598529 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j8hf\" (UniqueName: \"kubernetes.io/projected/8c390c71-420f-489f-9a84-0ac10e6a74ef-kube-api-access-8j8hf\") pod \"maas-api-revoke-sub-6d439241-mf8d2\" (UID: \"8c390c71-420f-489f-9a84-0ac10e6a74ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d439241-mf8d2" Oct 30 18:07:03.659311 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:03.659278 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:07:03.659473 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:03.659287 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\"" Oct 30 18:07:03.700169 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:03.700135 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c390c71-420f-489f-9a84-0ac10e6a74ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d439241-mf8d2\" (UID: \"8c390c71-420f-489f-9a84-0ac10e6a74ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d439241-mf8d2" Oct 30 18:07:03.700334 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:03.700228 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j8hf\" (UniqueName: \"kubernetes.io/projected/8c390c71-420f-489f-9a84-0ac10e6a74ef-kube-api-access-8j8hf\") pod \"maas-api-revoke-sub-6d439241-mf8d2\" (UID: \"8c390c71-420f-489f-9a84-0ac10e6a74ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d439241-mf8d2" Oct 30 18:07:03.700334 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:03.700327 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c390c71-420f-489f-9a84-0ac10e6a74ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d439241-mf8d2\" (UID: \"8c390c71-420f-489f-9a84-0ac10e6a74ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d439241-mf8d2" Oct 30 18:07:03.700943 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:03.700916 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c390c71-420f-489f-9a84-0ac10e6a74ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d439241-mf8d2\" (UID: \"8c390c71-420f-489f-9a84-0ac10e6a74ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d439241-mf8d2" Oct 30 18:07:03.702500 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:03.702481 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c390c71-420f-489f-9a84-0ac10e6a74ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d439241-mf8d2\" (UID: \"8c390c71-420f-489f-9a84-0ac10e6a74ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d439241-mf8d2" Oct 30 18:07:03.723579 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:03.723554 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j8hf\" (UniqueName: \"kubernetes.io/projected/8c390c71-420f-489f-9a84-0ac10e6a74ef-kube-api-access-8j8hf\") pod \"maas-api-revoke-sub-6d439241-mf8d2\" (UID: \"8c390c71-420f-489f-9a84-0ac10e6a74ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d439241-mf8d2" Oct 30 18:07:03.884414 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:03.884384 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d439241-mf8d2" Oct 30 18:07:04.197655 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:04.197621 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d439241-mf8d2"] Oct 30 18:07:04.199042 ip-10-0-139-222 kubenswrapper[4965]: W1030 18:07:04.199003 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c390c71_420f_489f_9a84_0ac10e6a74ef.slice/crio-491d6839f3e08ad45ab4a7d9f37ffebf9518f7280b94860394146733da088cac WatchSource:0}: Error finding container 491d6839f3e08ad45ab4a7d9f37ffebf9518f7280b94860394146733da088cac: Status 404 returned error can't find the container with id 491d6839f3e08ad45ab4a7d9f37ffebf9518f7280b94860394146733da088cac Oct 30 18:07:04.538750 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:04.538714 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d439241-mf8d2" event={"ID":"8c390c71-420f-489f-9a84-0ac10e6a74ef","Type":"ContainerStarted","Data":"17f480667741ef51803e733019ef1de6b2bca823b4002199418f1c1f460567b3"} Oct 30 18:07:04.538948 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:04.538766 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d439241-mf8d2" event={"ID":"8c390c71-420f-489f-9a84-0ac10e6a74ef","Type":"ContainerStarted","Data":"491d6839f3e08ad45ab4a7d9f37ffebf9518f7280b94860394146733da088cac"} Oct 30 18:07:04.581516 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:04.581452 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d439241-mf8d2" podStartSLOduration=1.581432199 podStartE2EDuration="1.581432199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:04.580165961 +0000 UTC m=+3282.632265892" watchObservedRunningTime="2026-10-30 18:07:04.581432199 +0000 UTC m=+3282.633532132" Oct 30 18:07:04.854918 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:04.854893 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-506124-59bc6cd6c4-rp7pn" Oct 30 18:07:04.914246 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:04.914201 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6540385c-e0cf-41cd-8e45-123e600e2038-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6540385c-e0cf-41cd-8e45-123e600e2038-plugins-config-volume\") pod \"6540385c-e0cf-41cd-8e45-123e600e2038\" (UID: \"6540385c-e0cf-41cd-8e45-123e600e2038\") " Oct 30 18:07:04.914440 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:04.914262 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6540385c-e0cf-41cd-8e45-123e600e2038-kube-api-access-b4scf\" (UniqueName: \"kubernetes.io/projected/6540385c-e0cf-41cd-8e45-123e600e2038-kube-api-access-b4scf\") pod \"6540385c-e0cf-41cd-8e45-123e600e2038\" (UID: \"6540385c-e0cf-41cd-8e45-123e600e2038\") " Oct 30 18:07:04.915019 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:04.914983 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6540385c-e0cf-41cd-8e45-123e600e2038-plugins-config-volume" pod "6540385c-e0cf-41cd-8e45-123e600e2038" (UID: "6540385c-e0cf-41cd-8e45-123e600e2038"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:07:04.917553 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:04.917524 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6540385c-e0cf-41cd-8e45-123e600e2038-kube-api-access-b4scf" pod "6540385c-e0cf-41cd-8e45-123e600e2038" (UID: "6540385c-e0cf-41cd-8e45-123e600e2038"). InnerVolumeSpecName "kube-api-access-b4scf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:07:05.016146 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:05.016075 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6540385c-e0cf-41cd-8e45-123e600e2038-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:05.016146 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:05.016100 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4scf\" (UniqueName: \"kubernetes.io/projected/6540385c-e0cf-41cd-8e45-123e600e2038-kube-api-access-b4scf\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:05.547519 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:05.547479 4965 generic.go:356] "Generic (PLEG): container finished" podID="8c390c71-420f-489f-9a84-0ac10e6a74ef" containerID="17f480667741ef51803e733019ef1de6b2bca823b4002199418f1c1f460567b3" exitCode=0 Oct 30 18:07:05.547742 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:05.547596 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d439241-mf8d2" event={"ID":"8c390c71-420f-489f-9a84-0ac10e6a74ef","Type":"ContainerDied","Data":"17f480667741ef51803e733019ef1de6b2bca823b4002199418f1c1f460567b3"} Oct 30 18:07:05.549031 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:05.549008 4965 generic.go:356] "Generic (PLEG): container finished" podID="6540385c-e0cf-41cd-8e45-123e600e2038" containerID="4db60a6605f95d02333f5748f3e1dbb409bd4cfd1cea30df1069105acaff950f" exitCode=0 Oct 30 18:07:05.549167 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:05.549050 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-506124-59bc6cd6c4-rp7pn" event={"ID":"6540385c-e0cf-41cd-8e45-123e600e2038","Type":"ContainerDied","Data":"4db60a6605f95d02333f5748f3e1dbb409bd4cfd1cea30df1069105acaff950f"} Oct 30 18:07:05.549167 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:05.549086 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-506124-59bc6cd6c4-rp7pn" event={"ID":"6540385c-e0cf-41cd-8e45-123e600e2038","Type":"ContainerDied","Data":"de64fa2a230f99f0974c77773d3fb532be8bb76acef2d9a82cdd024a8aeea329"} Oct 30 18:07:05.549167 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:05.549089 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-506124-59bc6cd6c4-rp7pn" Oct 30 18:07:05.549167 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:05.549116 4965 scope.go:122] "RemoveContainer" containerID="4db60a6605f95d02333f5748f3e1dbb409bd4cfd1cea30df1069105acaff950f" Oct 30 18:07:05.560655 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:05.560639 4965 scope.go:122] "RemoveContainer" containerID="4db60a6605f95d02333f5748f3e1dbb409bd4cfd1cea30df1069105acaff950f" Oct 30 18:07:05.560941 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:07:05.560911 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4db60a6605f95d02333f5748f3e1dbb409bd4cfd1cea30df1069105acaff950f\": container with ID starting with 4db60a6605f95d02333f5748f3e1dbb409bd4cfd1cea30df1069105acaff950f not found: ID does not exist" containerID="4db60a6605f95d02333f5748f3e1dbb409bd4cfd1cea30df1069105acaff950f" Oct 30 18:07:05.561033 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:05.560937 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4db60a6605f95d02333f5748f3e1dbb409bd4cfd1cea30df1069105acaff950f"} err="failed to get container status \"4db60a6605f95d02333f5748f3e1dbb409bd4cfd1cea30df1069105acaff950f\": rpc error: code = NotFound desc = could not find container \"4db60a6605f95d02333f5748f3e1dbb409bd4cfd1cea30df1069105acaff950f\": container with ID starting with 4db60a6605f95d02333f5748f3e1dbb409bd4cfd1cea30df1069105acaff950f not found: ID does not exist" Oct 30 18:07:05.637496 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:05.637459 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-506124-59bc6cd6c4-rp7pn"] Oct 30 18:07:05.642369 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:05.642337 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-506124-59bc6cd6c4-rp7pn"] Oct 30 18:07:05.997145 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:07:05.997108 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:07:06.098545 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.098513 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6540385c-e0cf-41cd-8e45-123e600e2038" path="/var/lib/kubelet/pods/6540385c-e0cf-41cd-8e45-123e600e2038/volumes" Oct 30 18:07:06.383578 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.383554 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 18:07:06.433087 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.433060 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8f96554-6aec-4052-8219-307fb11b6404-kube-api-access-f46hz\" (UniqueName: \"kubernetes.io/projected/d8f96554-6aec-4052-8219-307fb11b6404-kube-api-access-f46hz\") pod \"d8f96554-6aec-4052-8219-307fb11b6404\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " Oct 30 18:07:06.433235 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.433136 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8f96554-6aec-4052-8219-307fb11b6404-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8f96554-6aec-4052-8219-307fb11b6404-istio-podinfo\") pod \"d8f96554-6aec-4052-8219-307fb11b6404\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " Oct 30 18:07:06.433235 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.433180 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8f96554-6aec-4052-8219-307fb11b6404-istio-token\" (UniqueName: \"kubernetes.io/projected/d8f96554-6aec-4052-8219-307fb11b6404-istio-token\") pod \"d8f96554-6aec-4052-8219-307fb11b6404\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " Oct 30 18:07:06.433326 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.433262 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-credential-socket\") pod \"d8f96554-6aec-4052-8219-307fb11b6404\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " Oct 30 18:07:06.433326 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.433307 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-workload-socket\") pod \"d8f96554-6aec-4052-8219-307fb11b6404\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " Oct 30 18:07:06.433422 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.433342 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-workload-certs\") pod \"d8f96554-6aec-4052-8219-307fb11b6404\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " Oct 30 18:07:06.433422 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.433368 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-istio-data\") pod \"d8f96554-6aec-4052-8219-307fb11b6404\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " Oct 30 18:07:06.433422 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.433402 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8f96554-6aec-4052-8219-307fb11b6404-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8f96554-6aec-4052-8219-307fb11b6404-istiod-ca-cert\") pod \"d8f96554-6aec-4052-8219-307fb11b6404\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " Oct 30 18:07:06.433597 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.433501 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-istio-envoy\") pod \"d8f96554-6aec-4052-8219-307fb11b6404\" (UID: \"d8f96554-6aec-4052-8219-307fb11b6404\") " Oct 30 18:07:06.433655 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.433577 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-credential-socket" pod "d8f96554-6aec-4052-8219-307fb11b6404" (UID: "d8f96554-6aec-4052-8219-307fb11b6404"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:07:06.433655 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.433623 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-workload-socket" pod "d8f96554-6aec-4052-8219-307fb11b6404" (UID: "d8f96554-6aec-4052-8219-307fb11b6404"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:07:06.433797 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.433758 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-istio-data" pod "d8f96554-6aec-4052-8219-307fb11b6404" (UID: "d8f96554-6aec-4052-8219-307fb11b6404"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:07:06.433954 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.433927 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8f96554-6aec-4052-8219-307fb11b6404-istiod-ca-cert" pod "d8f96554-6aec-4052-8219-307fb11b6404" (UID: "d8f96554-6aec-4052-8219-307fb11b6404"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:07:06.434090 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.434060 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:06.434090 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.434082 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:06.434263 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.434099 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:06.434263 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.434113 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8f96554-6aec-4052-8219-307fb11b6404-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:06.434263 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.434251 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-workload-certs" pod "d8f96554-6aec-4052-8219-307fb11b6404" (UID: "d8f96554-6aec-4052-8219-307fb11b6404"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:07:06.435467 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.435444 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8f96554-6aec-4052-8219-307fb11b6404-kube-api-access-f46hz" pod "d8f96554-6aec-4052-8219-307fb11b6404" (UID: "d8f96554-6aec-4052-8219-307fb11b6404"). InnerVolumeSpecName "kube-api-access-f46hz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:07:06.435574 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.435478 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8f96554-6aec-4052-8219-307fb11b6404-istio-podinfo" pod "d8f96554-6aec-4052-8219-307fb11b6404" (UID: "d8f96554-6aec-4052-8219-307fb11b6404"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:07:06.435574 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.435529 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8f96554-6aec-4052-8219-307fb11b6404-istio-token" pod "d8f96554-6aec-4052-8219-307fb11b6404" (UID: "d8f96554-6aec-4052-8219-307fb11b6404"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:07:06.435665 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.435637 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-istio-envoy" pod "d8f96554-6aec-4052-8219-307fb11b6404" (UID: "d8f96554-6aec-4052-8219-307fb11b6404"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:07:06.535172 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.535100 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:06.535172 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.535128 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f46hz\" (UniqueName: \"kubernetes.io/projected/d8f96554-6aec-4052-8219-307fb11b6404-kube-api-access-f46hz\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:06.535172 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.535139 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8f96554-6aec-4052-8219-307fb11b6404-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:06.535172 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.535147 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8f96554-6aec-4052-8219-307fb11b6404-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:06.535172 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.535156 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8f96554-6aec-4052-8219-307fb11b6404-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:06.557740 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.557703 4965 generic.go:356] "Generic (PLEG): container finished" podID="d8f96554-6aec-4052-8219-307fb11b6404" containerID="9c65aa992d826ddf4bd112dc532b7cded0a020ecf963add7b2cb0d8d69d2ce3d" exitCode=0 Oct 30 18:07:06.557888 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.557757 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" Oct 30 18:07:06.557888 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.557785 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" event={"ID":"d8f96554-6aec-4052-8219-307fb11b6404","Type":"ContainerDied","Data":"9c65aa992d826ddf4bd112dc532b7cded0a020ecf963add7b2cb0d8d69d2ce3d"} Oct 30 18:07:06.557888 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.557837 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5" event={"ID":"d8f96554-6aec-4052-8219-307fb11b6404","Type":"ContainerDied","Data":"00e4fb6dcb3e40e2761b4ef50b529ced3d2be7400034676173fde16b1664fca4"} Oct 30 18:07:06.557888 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.557882 4965 scope.go:122] "RemoveContainer" containerID="9c65aa992d826ddf4bd112dc532b7cded0a020ecf963add7b2cb0d8d69d2ce3d" Oct 30 18:07:06.568232 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.568209 4965 scope.go:122] "RemoveContainer" containerID="9c65aa992d826ddf4bd112dc532b7cded0a020ecf963add7b2cb0d8d69d2ce3d" Oct 30 18:07:06.568578 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:07:06.568555 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c65aa992d826ddf4bd112dc532b7cded0a020ecf963add7b2cb0d8d69d2ce3d\": container with ID starting with 9c65aa992d826ddf4bd112dc532b7cded0a020ecf963add7b2cb0d8d69d2ce3d not found: ID does not exist" containerID="9c65aa992d826ddf4bd112dc532b7cded0a020ecf963add7b2cb0d8d69d2ce3d" Oct 30 18:07:06.568684 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.568585 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c65aa992d826ddf4bd112dc532b7cded0a020ecf963add7b2cb0d8d69d2ce3d"} err="failed to get container status \"9c65aa992d826ddf4bd112dc532b7cded0a020ecf963add7b2cb0d8d69d2ce3d\": rpc error: code = NotFound desc = could not find container \"9c65aa992d826ddf4bd112dc532b7cded0a020ecf963add7b2cb0d8d69d2ce3d\": container with ID starting with 9c65aa992d826ddf4bd112dc532b7cded0a020ecf963add7b2cb0d8d69d2ce3d not found: ID does not exist" Oct 30 18:07:06.614175 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.614125 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5"] Oct 30 18:07:06.622008 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.620281 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-d66d5"] Oct 30 18:07:06.696272 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.696248 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d439241-mf8d2" Oct 30 18:07:06.737443 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.737406 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c390c71-420f-489f-9a84-0ac10e6a74ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c390c71-420f-489f-9a84-0ac10e6a74ef-maas-api-openshift-ca\") pod \"8c390c71-420f-489f-9a84-0ac10e6a74ef\" (UID: \"8c390c71-420f-489f-9a84-0ac10e6a74ef\") " Oct 30 18:07:06.737655 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.737458 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c390c71-420f-489f-9a84-0ac10e6a74ef-kube-api-access-8j8hf\" (UniqueName: \"kubernetes.io/projected/8c390c71-420f-489f-9a84-0ac10e6a74ef-kube-api-access-8j8hf\") pod \"8c390c71-420f-489f-9a84-0ac10e6a74ef\" (UID: \"8c390c71-420f-489f-9a84-0ac10e6a74ef\") " Oct 30 18:07:06.737655 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.737580 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c390c71-420f-489f-9a84-0ac10e6a74ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c390c71-420f-489f-9a84-0ac10e6a74ef-maas-api-xks-ca\") pod \"8c390c71-420f-489f-9a84-0ac10e6a74ef\" (UID: \"8c390c71-420f-489f-9a84-0ac10e6a74ef\") " Oct 30 18:07:06.737903 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.737848 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c390c71-420f-489f-9a84-0ac10e6a74ef-maas-api-openshift-ca" pod "8c390c71-420f-489f-9a84-0ac10e6a74ef" (UID: "8c390c71-420f-489f-9a84-0ac10e6a74ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:07:06.738037 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.738021 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c390c71-420f-489f-9a84-0ac10e6a74ef-maas-api-openshift-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:06.739689 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.739665 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c390c71-420f-489f-9a84-0ac10e6a74ef-kube-api-access-8j8hf" pod "8c390c71-420f-489f-9a84-0ac10e6a74ef" (UID: "8c390c71-420f-489f-9a84-0ac10e6a74ef"). InnerVolumeSpecName "kube-api-access-8j8hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:07:06.739795 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.739702 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c390c71-420f-489f-9a84-0ac10e6a74ef-maas-api-xks-ca" pod "8c390c71-420f-489f-9a84-0ac10e6a74ef" (UID: "8c390c71-420f-489f-9a84-0ac10e6a74ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:07:06.839208 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.839176 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j8hf\" (UniqueName: \"kubernetes.io/projected/8c390c71-420f-489f-9a84-0ac10e6a74ef-kube-api-access-8j8hf\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:06.839208 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:06.839202 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c390c71-420f-489f-9a84-0ac10e6a74ef-maas-api-xks-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:07.565832 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:07.565747 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d439241-mf8d2" Oct 30 18:07:07.565832 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:07.565779 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d439241-mf8d2" event={"ID":"8c390c71-420f-489f-9a84-0ac10e6a74ef","Type":"ContainerDied","Data":"491d6839f3e08ad45ab4a7d9f37ffebf9518f7280b94860394146733da088cac"} Oct 30 18:07:07.565832 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:07.565823 4965 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="491d6839f3e08ad45ab4a7d9f37ffebf9518f7280b94860394146733da088cac" Oct 30 18:07:07.652639 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:07.652608 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:07:08.098494 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:08.098462 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8f96554-6aec-4052-8219-307fb11b6404" path="/var/lib/kubelet/pods/d8f96554-6aec-4052-8219-307fb11b6404/volumes" Oct 30 18:07:08.855838 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:08.855804 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz"] Oct 30 18:07:08.856078 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:08.856046 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz" podUID="3e25504b-8631-4770-9d29-a0ee77fd9116" containerName="maas-api" containerID="cri-o://d5c7d0923ed7610a00d19bc664cfdd2f655aee9d05a66ee2a5e7ef5b476f3acb" gracePeriod=30 Oct 30 18:07:09.180972 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:09.180950 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz" Oct 30 18:07:09.318759 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:09.318720 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e25504b-8631-4770-9d29-a0ee77fd9116-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e25504b-8631-4770-9d29-a0ee77fd9116-maas-api-tls\") pod \"3e25504b-8631-4770-9d29-a0ee77fd9116\" (UID: \"3e25504b-8631-4770-9d29-a0ee77fd9116\") " Oct 30 18:07:09.318988 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:09.318927 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e25504b-8631-4770-9d29-a0ee77fd9116-kube-api-access-ffz99\" (UniqueName: \"kubernetes.io/projected/3e25504b-8631-4770-9d29-a0ee77fd9116-kube-api-access-ffz99\") pod \"3e25504b-8631-4770-9d29-a0ee77fd9116\" (UID: \"3e25504b-8631-4770-9d29-a0ee77fd9116\") " Oct 30 18:07:09.319067 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:09.319032 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e25504b-8631-4770-9d29-a0ee77fd9116-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e25504b-8631-4770-9d29-a0ee77fd9116-openshift-service-ca\") pod \"3e25504b-8631-4770-9d29-a0ee77fd9116\" (UID: \"3e25504b-8631-4770-9d29-a0ee77fd9116\") " Oct 30 18:07:09.319249 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:09.319207 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e25504b-8631-4770-9d29-a0ee77fd9116-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e25504b-8631-4770-9d29-a0ee77fd9116-metrics-certs\") pod \"3e25504b-8631-4770-9d29-a0ee77fd9116\" (UID: \"3e25504b-8631-4770-9d29-a0ee77fd9116\") " Oct 30 18:07:09.319511 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:09.319484 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e25504b-8631-4770-9d29-a0ee77fd9116-openshift-service-ca" pod "3e25504b-8631-4770-9d29-a0ee77fd9116" (UID: "3e25504b-8631-4770-9d29-a0ee77fd9116"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:07:09.320255 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:09.320232 4965 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e25504b-8631-4770-9d29-a0ee77fd9116-openshift-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:09.321939 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:09.321746 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e25504b-8631-4770-9d29-a0ee77fd9116-metrics-certs" pod "3e25504b-8631-4770-9d29-a0ee77fd9116" (UID: "3e25504b-8631-4770-9d29-a0ee77fd9116"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:07:09.321939 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:09.321761 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e25504b-8631-4770-9d29-a0ee77fd9116-maas-api-tls" pod "3e25504b-8631-4770-9d29-a0ee77fd9116" (UID: "3e25504b-8631-4770-9d29-a0ee77fd9116"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:07:09.322059 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:09.322002 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e25504b-8631-4770-9d29-a0ee77fd9116-kube-api-access-ffz99" pod "3e25504b-8631-4770-9d29-a0ee77fd9116" (UID: "3e25504b-8631-4770-9d29-a0ee77fd9116"). InnerVolumeSpecName "kube-api-access-ffz99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:07:09.421724 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:09.421647 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffz99\" (UniqueName: \"kubernetes.io/projected/3e25504b-8631-4770-9d29-a0ee77fd9116-kube-api-access-ffz99\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:09.421724 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:09.421688 4965 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e25504b-8631-4770-9d29-a0ee77fd9116-metrics-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:09.421724 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:09.421704 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e25504b-8631-4770-9d29-a0ee77fd9116-maas-api-tls\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:09.598706 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:09.598670 4965 generic.go:356] "Generic (PLEG): container finished" podID="3e25504b-8631-4770-9d29-a0ee77fd9116" containerID="d5c7d0923ed7610a00d19bc664cfdd2f655aee9d05a66ee2a5e7ef5b476f3acb" exitCode=0 Oct 30 18:07:09.598927 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:09.598733 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz" Oct 30 18:07:09.598927 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:09.598782 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz" event={"ID":"3e25504b-8631-4770-9d29-a0ee77fd9116","Type":"ContainerDied","Data":"d5c7d0923ed7610a00d19bc664cfdd2f655aee9d05a66ee2a5e7ef5b476f3acb"} Oct 30 18:07:09.598927 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:09.598852 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz" event={"ID":"3e25504b-8631-4770-9d29-a0ee77fd9116","Type":"ContainerDied","Data":"d50b4cb1986e4320f5b76254162611515338521857bc798091948aa2b3677cce"} Oct 30 18:07:09.598927 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:09.598891 4965 scope.go:122] "RemoveContainer" containerID="d5c7d0923ed7610a00d19bc664cfdd2f655aee9d05a66ee2a5e7ef5b476f3acb" Oct 30 18:07:09.611377 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:09.611049 4965 scope.go:122] "RemoveContainer" containerID="d5c7d0923ed7610a00d19bc664cfdd2f655aee9d05a66ee2a5e7ef5b476f3acb" Oct 30 18:07:09.611548 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:07:09.611490 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5c7d0923ed7610a00d19bc664cfdd2f655aee9d05a66ee2a5e7ef5b476f3acb\": container with ID starting with d5c7d0923ed7610a00d19bc664cfdd2f655aee9d05a66ee2a5e7ef5b476f3acb not found: ID does not exist" containerID="d5c7d0923ed7610a00d19bc664cfdd2f655aee9d05a66ee2a5e7ef5b476f3acb" Oct 30 18:07:09.611548 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:09.611524 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c7d0923ed7610a00d19bc664cfdd2f655aee9d05a66ee2a5e7ef5b476f3acb"} err="failed to get container status \"d5c7d0923ed7610a00d19bc664cfdd2f655aee9d05a66ee2a5e7ef5b476f3acb\": rpc error: code = NotFound desc = could not find container \"d5c7d0923ed7610a00d19bc664cfdd2f655aee9d05a66ee2a5e7ef5b476f3acb\": container with ID starting with d5c7d0923ed7610a00d19bc664cfdd2f655aee9d05a66ee2a5e7ef5b476f3acb not found: ID does not exist" Oct 30 18:07:09.667544 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:09.667501 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz"] Oct 30 18:07:09.673395 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:09.673323 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-506124-86b5665b4d-nwgpz"] Oct 30 18:07:10.098195 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:10.098162 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e25504b-8631-4770-9d29-a0ee77fd9116" path="/var/lib/kubelet/pods/3e25504b-8631-4770-9d29-a0ee77fd9116/volumes" Oct 30 18:07:16.380295 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:16.380258 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96c514-5cd48b54dd-8tn7q"] Oct 30 18:07:16.380779 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:16.380465 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-5cd48b54dd-8tn7q" podUID="42a77847-5a9e-4825-80db-5f0b19133f62" containerName="payload-processing" containerID="cri-o://a10739ec5ad178afa0aa36a86727d7b064b18c5021d688377638ede7d034c1c1" gracePeriod=30 Oct 30 18:07:16.432892 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:16.426812 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-6dcd6678f9-jjjkd"] Oct 30 18:07:16.433407 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:16.433310 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-6dcd6678f9-jjjkd" podUID="2c52d606-3e47-4e08-bb5d-4f018e638134" containerName="payload-pre-processing" containerID="cri-o://de5f4896d5fcd8288631835d0a90ab7d2375f0f6b05df8f5aaa00b547f323a38" gracePeriod=30 Oct 30 18:07:16.649399 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:16.649316 4965 generic.go:356] "Generic (PLEG): container finished" podID="42a77847-5a9e-4825-80db-5f0b19133f62" containerID="a10739ec5ad178afa0aa36a86727d7b064b18c5021d688377638ede7d034c1c1" exitCode=0 Oct 30 18:07:16.649564 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:16.649456 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-5cd48b54dd-8tn7q" event={"ID":"42a77847-5a9e-4825-80db-5f0b19133f62","Type":"ContainerDied","Data":"a10739ec5ad178afa0aa36a86727d7b064b18c5021d688377638ede7d034c1c1"} Oct 30 18:07:16.651611 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:16.651580 4965 generic.go:356] "Generic (PLEG): container finished" podID="2c52d606-3e47-4e08-bb5d-4f018e638134" containerID="de5f4896d5fcd8288631835d0a90ab7d2375f0f6b05df8f5aaa00b547f323a38" exitCode=0 Oct 30 18:07:16.651757 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:16.651713 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-6dcd6678f9-jjjkd" event={"ID":"2c52d606-3e47-4e08-bb5d-4f018e638134","Type":"ContainerDied","Data":"de5f4896d5fcd8288631835d0a90ab7d2375f0f6b05df8f5aaa00b547f323a38"} Oct 30 18:07:16.800223 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:16.800197 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-5cd48b54dd-8tn7q" Oct 30 18:07:16.802439 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:16.802420 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-6dcd6678f9-jjjkd" Oct 30 18:07:16.931237 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:16.931151 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42a77847-5a9e-4825-80db-5f0b19133f62-kube-api-access-k8m85\" (UniqueName: \"kubernetes.io/projected/42a77847-5a9e-4825-80db-5f0b19133f62-kube-api-access-k8m85\") pod \"42a77847-5a9e-4825-80db-5f0b19133f62\" (UID: \"42a77847-5a9e-4825-80db-5f0b19133f62\") " Oct 30 18:07:16.931237 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:16.931200 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42a77847-5a9e-4825-80db-5f0b19133f62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42a77847-5a9e-4825-80db-5f0b19133f62-plugins-config-volume\") pod \"42a77847-5a9e-4825-80db-5f0b19133f62\" (UID: \"42a77847-5a9e-4825-80db-5f0b19133f62\") " Oct 30 18:07:16.931491 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:16.931276 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c52d606-3e47-4e08-bb5d-4f018e638134-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c52d606-3e47-4e08-bb5d-4f018e638134-plugins-config-volume\") pod \"2c52d606-3e47-4e08-bb5d-4f018e638134\" (UID: \"2c52d606-3e47-4e08-bb5d-4f018e638134\") " Oct 30 18:07:16.931491 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:16.931434 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c52d606-3e47-4e08-bb5d-4f018e638134-kube-api-access-wh8kr\" (UniqueName: \"kubernetes.io/projected/2c52d606-3e47-4e08-bb5d-4f018e638134-kube-api-access-wh8kr\") pod \"2c52d606-3e47-4e08-bb5d-4f018e638134\" (UID: \"2c52d606-3e47-4e08-bb5d-4f018e638134\") " Oct 30 18:07:16.931694 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:16.931671 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42a77847-5a9e-4825-80db-5f0b19133f62-plugins-config-volume" pod "42a77847-5a9e-4825-80db-5f0b19133f62" (UID: "42a77847-5a9e-4825-80db-5f0b19133f62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:07:16.931772 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:16.931671 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c52d606-3e47-4e08-bb5d-4f018e638134-plugins-config-volume" pod "2c52d606-3e47-4e08-bb5d-4f018e638134" (UID: "2c52d606-3e47-4e08-bb5d-4f018e638134"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:07:16.931923 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:16.931906 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42a77847-5a9e-4825-80db-5f0b19133f62-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:16.931975 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:16.931926 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c52d606-3e47-4e08-bb5d-4f018e638134-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:16.933404 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:16.933387 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a77847-5a9e-4825-80db-5f0b19133f62-kube-api-access-k8m85" pod "42a77847-5a9e-4825-80db-5f0b19133f62" (UID: "42a77847-5a9e-4825-80db-5f0b19133f62"). InnerVolumeSpecName "kube-api-access-k8m85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:07:16.933921 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:16.933900 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c52d606-3e47-4e08-bb5d-4f018e638134-kube-api-access-wh8kr" pod "2c52d606-3e47-4e08-bb5d-4f018e638134" (UID: "2c52d606-3e47-4e08-bb5d-4f018e638134"). InnerVolumeSpecName "kube-api-access-wh8kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:07:17.033439 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:17.033405 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8m85\" (UniqueName: \"kubernetes.io/projected/42a77847-5a9e-4825-80db-5f0b19133f62-kube-api-access-k8m85\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:17.033439 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:17.033437 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh8kr\" (UniqueName: \"kubernetes.io/projected/2c52d606-3e47-4e08-bb5d-4f018e638134-kube-api-access-wh8kr\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:17.284404 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:17.284318 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4"] Oct 30 18:07:17.284666 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:17.284632 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" podUID="ddb82e7c-5038-46b7-937e-c9ae4b12b2c0" containerName="istio-proxy" containerID="cri-o://7a6434102ff2f3ea0f60364af178fb96ef8e689579d38dcf8e72684777f76f3f" gracePeriod=30 Oct 30 18:07:17.658754 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:17.658726 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-5cd48b54dd-8tn7q" Oct 30 18:07:17.659276 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:17.658768 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-96c514-5cd48b54dd-8tn7q" event={"ID":"42a77847-5a9e-4825-80db-5f0b19133f62","Type":"ContainerDied","Data":"41f221d93adaaab55a6a2d656b41594243be7b507643fb4dfbbf8d2a949117a1"} Oct 30 18:07:17.659276 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:17.658835 4965 scope.go:122] "RemoveContainer" containerID="a10739ec5ad178afa0aa36a86727d7b064b18c5021d688377638ede7d034c1c1" Oct 30 18:07:17.660394 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:17.660366 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-6dcd6678f9-jjjkd" Oct 30 18:07:17.660474 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:17.660373 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-6dcd6678f9-jjjkd" event={"ID":"2c52d606-3e47-4e08-bb5d-4f018e638134","Type":"ContainerDied","Data":"4431e4829532fdaa772f7ce75670e5a4235c6deab1215a4cf4c837f2cb53f51e"} Oct 30 18:07:17.668740 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:17.668722 4965 scope.go:122] "RemoveContainer" containerID="de5f4896d5fcd8288631835d0a90ab7d2375f0f6b05df8f5aaa00b547f323a38" Oct 30 18:07:17.712608 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:17.712582 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96c514-5cd48b54dd-8tn7q"] Oct 30 18:07:17.724589 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:17.724565 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96c514-5cd48b54dd-8tn7q"] Oct 30 18:07:17.747040 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:17.747013 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-6dcd6678f9-jjjkd"] Oct 30 18:07:17.751811 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:17.751785 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96c514-6dcd6678f9-jjjkd"] Oct 30 18:07:18.098038 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:18.098006 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c52d606-3e47-4e08-bb5d-4f018e638134" path="/var/lib/kubelet/pods/2c52d606-3e47-4e08-bb5d-4f018e638134/volumes" Oct 30 18:07:18.098534 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:18.098512 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42a77847-5a9e-4825-80db-5f0b19133f62" path="/var/lib/kubelet/pods/42a77847-5a9e-4825-80db-5f0b19133f62/volumes" Oct 30 18:07:19.927475 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:19.927410 4965 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:07:19.927974 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:19.927467 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" podUID="ddb82e7c-5038-46b7-937e-c9ae4b12b2c0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:07:22.596530 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.596505 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 18:07:22.697738 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.697699 4965 generic.go:356] "Generic (PLEG): container finished" podID="ddb82e7c-5038-46b7-937e-c9ae4b12b2c0" containerID="7a6434102ff2f3ea0f60364af178fb96ef8e689579d38dcf8e72684777f76f3f" exitCode=0 Oct 30 18:07:22.697927 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.697756 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" Oct 30 18:07:22.697927 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.697777 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" event={"ID":"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0","Type":"ContainerDied","Data":"7a6434102ff2f3ea0f60364af178fb96ef8e689579d38dcf8e72684777f76f3f"} Oct 30 18:07:22.697927 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.697847 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4" event={"ID":"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0","Type":"ContainerDied","Data":"4db416214e7e09eeb42c38fe4081ef2108444f8e2bf22fa962dfbbe96533702a"} Oct 30 18:07:22.697927 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.697906 4965 scope.go:122] "RemoveContainer" containerID="7a6434102ff2f3ea0f60364af178fb96ef8e689579d38dcf8e72684777f76f3f" Oct 30 18:07:22.706445 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.706419 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-workload-certs\") pod \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " Oct 30 18:07:22.706597 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.706468 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istio-token\" (UniqueName: \"kubernetes.io/projected/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istio-token\") pod \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " Oct 30 18:07:22.706597 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.706531 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-credential-socket\") pod \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " Oct 30 18:07:22.706597 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.706553 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istio-envoy\") pod \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " Oct 30 18:07:22.706822 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.706610 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istiod-ca-cert\") pod \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " Oct 30 18:07:22.706822 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.706645 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istio-data\") pod \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " Oct 30 18:07:22.706822 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.706673 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-kube-api-access-nqtpf\" (UniqueName: \"kubernetes.io/projected/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-kube-api-access-nqtpf\") pod \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " Oct 30 18:07:22.706822 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.706693 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istio-podinfo\") pod \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " Oct 30 18:07:22.706822 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.706694 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-workload-certs" pod "ddb82e7c-5038-46b7-937e-c9ae4b12b2c0" (UID: "ddb82e7c-5038-46b7-937e-c9ae4b12b2c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:07:22.706822 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.706713 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-workload-socket\") pod \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\" (UID: \"ddb82e7c-5038-46b7-937e-c9ae4b12b2c0\") " Oct 30 18:07:22.707179 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.706984 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istiod-ca-cert" pod "ddb82e7c-5038-46b7-937e-c9ae4b12b2c0" (UID: "ddb82e7c-5038-46b7-937e-c9ae4b12b2c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:07:22.707179 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.707091 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istio-data" pod "ddb82e7c-5038-46b7-937e-c9ae4b12b2c0" (UID: "ddb82e7c-5038-46b7-937e-c9ae4b12b2c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:07:22.707276 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.707212 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:22.707276 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.707231 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:22.707276 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.707236 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-workload-socket" pod "ddb82e7c-5038-46b7-937e-c9ae4b12b2c0" (UID: "ddb82e7c-5038-46b7-937e-c9ae4b12b2c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:07:22.707276 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.707245 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:22.707276 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.707255 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-credential-socket" pod "ddb82e7c-5038-46b7-937e-c9ae4b12b2c0" (UID: "ddb82e7c-5038-46b7-937e-c9ae4b12b2c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:07:22.709250 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.709224 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istio-token" pod "ddb82e7c-5038-46b7-937e-c9ae4b12b2c0" (UID: "ddb82e7c-5038-46b7-937e-c9ae4b12b2c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:07:22.709555 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.709533 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-kube-api-access-nqtpf" pod "ddb82e7c-5038-46b7-937e-c9ae4b12b2c0" (UID: "ddb82e7c-5038-46b7-937e-c9ae4b12b2c0"). InnerVolumeSpecName "kube-api-access-nqtpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:07:22.709672 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.709540 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istio-envoy" pod "ddb82e7c-5038-46b7-937e-c9ae4b12b2c0" (UID: "ddb82e7c-5038-46b7-937e-c9ae4b12b2c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:07:22.709792 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.709756 4965 scope.go:122] "RemoveContainer" containerID="88b179d4d49aef2174d85adf10dae147bf57f623bdfb217c8831e1460c2fe2c2" Oct 30 18:07:22.715117 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.715093 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istio-podinfo" pod "ddb82e7c-5038-46b7-937e-c9ae4b12b2c0" (UID: "ddb82e7c-5038-46b7-937e-c9ae4b12b2c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:07:22.733200 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.733182 4965 scope.go:122] "RemoveContainer" containerID="7a6434102ff2f3ea0f60364af178fb96ef8e689579d38dcf8e72684777f76f3f" Oct 30 18:07:22.733496 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:07:22.733479 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a6434102ff2f3ea0f60364af178fb96ef8e689579d38dcf8e72684777f76f3f\": container with ID starting with 7a6434102ff2f3ea0f60364af178fb96ef8e689579d38dcf8e72684777f76f3f not found: ID does not exist" containerID="7a6434102ff2f3ea0f60364af178fb96ef8e689579d38dcf8e72684777f76f3f" Oct 30 18:07:22.733573 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.733504 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a6434102ff2f3ea0f60364af178fb96ef8e689579d38dcf8e72684777f76f3f"} err="failed to get container status \"7a6434102ff2f3ea0f60364af178fb96ef8e689579d38dcf8e72684777f76f3f\": rpc error: code = NotFound desc = could not find container \"7a6434102ff2f3ea0f60364af178fb96ef8e689579d38dcf8e72684777f76f3f\": container with ID starting with 7a6434102ff2f3ea0f60364af178fb96ef8e689579d38dcf8e72684777f76f3f not found: ID does not exist" Oct 30 18:07:22.733573 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.733520 4965 scope.go:122] "RemoveContainer" containerID="88b179d4d49aef2174d85adf10dae147bf57f623bdfb217c8831e1460c2fe2c2" Oct 30 18:07:22.733750 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:07:22.733734 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88b179d4d49aef2174d85adf10dae147bf57f623bdfb217c8831e1460c2fe2c2\": container with ID starting with 88b179d4d49aef2174d85adf10dae147bf57f623bdfb217c8831e1460c2fe2c2 not found: ID does not exist" containerID="88b179d4d49aef2174d85adf10dae147bf57f623bdfb217c8831e1460c2fe2c2" Oct 30 18:07:22.733792 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.733755 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88b179d4d49aef2174d85adf10dae147bf57f623bdfb217c8831e1460c2fe2c2"} err="failed to get container status \"88b179d4d49aef2174d85adf10dae147bf57f623bdfb217c8831e1460c2fe2c2\": rpc error: code = NotFound desc = could not find container \"88b179d4d49aef2174d85adf10dae147bf57f623bdfb217c8831e1460c2fe2c2\": container with ID starting with 88b179d4d49aef2174d85adf10dae147bf57f623bdfb217c8831e1460c2fe2c2 not found: ID does not exist" Oct 30 18:07:22.808709 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.808674 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:22.808709 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.808706 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:22.808919 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.808721 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:22.808919 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.808735 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqtpf\" (UniqueName: \"kubernetes.io/projected/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-kube-api-access-nqtpf\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:22.808919 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.808745 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:22.808919 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.808753 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:22.865524 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.865494 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 18:07:22.869764 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:22.869736 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 18:07:23.043719 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:23.043679 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4"] Oct 30 18:07:23.055308 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:23.055277 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-6kbz4"] Oct 30 18:07:24.103358 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:24.103327 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddb82e7c-5038-46b7-937e-c9ae4b12b2c0" path="/var/lib/kubelet/pods/ddb82e7c-5038-46b7-937e-c9ae4b12b2c0/volumes" Oct 30 18:07:33.037811 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:33.037772 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz"] Oct 30 18:07:33.038319 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:33.038092 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" podUID="d4d5fb88-e105-459f-a9cd-959eb131c883" containerName="istio-proxy" containerID="cri-o://3dc003052fc409784cd6968b3a1625432af4248d4b15dd368a7618d27e3adda8" gracePeriod=30 Oct 30 18:07:35.234903 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.234853 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7"] Oct 30 18:07:35.235506 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.235484 4965 cpu_manager.go:414] "containerMap: removing container" podUID="8c390c71-420f-489f-9a84-0ac10e6a74ef" containerName="revoke-keys" Oct 30 18:07:35.235598 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.235506 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c390c71-420f-489f-9a84-0ac10e6a74ef" containerName="revoke-keys" Oct 30 18:07:35.235598 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.235521 4965 cpu_manager.go:414] "containerMap: removing container" podUID="ddb82e7c-5038-46b7-937e-c9ae4b12b2c0" containerName="istio-proxy" Oct 30 18:07:35.235598 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.235526 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddb82e7c-5038-46b7-937e-c9ae4b12b2c0" containerName="istio-proxy" Oct 30 18:07:35.235598 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.235544 4965 cpu_manager.go:414] "containerMap: removing container" podUID="d8f96554-6aec-4052-8219-307fb11b6404" containerName="istio-proxy" Oct 30 18:07:35.235598 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.235549 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8f96554-6aec-4052-8219-307fb11b6404" containerName="istio-proxy" Oct 30 18:07:35.235598 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.235560 4965 cpu_manager.go:414] "containerMap: removing container" podUID="3e25504b-8631-4770-9d29-a0ee77fd9116" containerName="maas-api" Oct 30 18:07:35.235598 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.235566 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e25504b-8631-4770-9d29-a0ee77fd9116" containerName="maas-api" Oct 30 18:07:35.235598 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.235581 4965 cpu_manager.go:414] "containerMap: removing container" podUID="2c52d606-3e47-4e08-bb5d-4f018e638134" containerName="payload-pre-processing" Oct 30 18:07:35.235598 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.235587 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c52d606-3e47-4e08-bb5d-4f018e638134" containerName="payload-pre-processing" Oct 30 18:07:35.235598 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.235603 4965 cpu_manager.go:414] "containerMap: removing container" podUID="42a77847-5a9e-4825-80db-5f0b19133f62" containerName="payload-processing" Oct 30 18:07:35.236153 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.235607 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42a77847-5a9e-4825-80db-5f0b19133f62" containerName="payload-processing" Oct 30 18:07:35.236153 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.235618 4965 cpu_manager.go:414] "containerMap: removing container" podUID="ddb82e7c-5038-46b7-937e-c9ae4b12b2c0" containerName="istio-proxy" Oct 30 18:07:35.236153 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.235624 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddb82e7c-5038-46b7-937e-c9ae4b12b2c0" containerName="istio-proxy" Oct 30 18:07:35.236153 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.235652 4965 cpu_manager.go:414] "containerMap: removing container" podUID="6540385c-e0cf-41cd-8e45-123e600e2038" containerName="payload-processing" Oct 30 18:07:35.236153 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.235659 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6540385c-e0cf-41cd-8e45-123e600e2038" containerName="payload-processing" Oct 30 18:07:35.236153 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.235777 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="6540385c-e0cf-41cd-8e45-123e600e2038" containerName="payload-processing" Oct 30 18:07:35.236153 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.235791 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c390c71-420f-489f-9a84-0ac10e6a74ef" containerName="revoke-keys" Oct 30 18:07:35.236153 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.235798 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddb82e7c-5038-46b7-937e-c9ae4b12b2c0" containerName="istio-proxy" Oct 30 18:07:35.236153 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.235810 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8f96554-6aec-4052-8219-307fb11b6404" containerName="istio-proxy" Oct 30 18:07:35.236153 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.235821 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e25504b-8631-4770-9d29-a0ee77fd9116" containerName="maas-api" Oct 30 18:07:35.236153 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.235828 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c52d606-3e47-4e08-bb5d-4f018e638134" containerName="payload-pre-processing" Oct 30 18:07:35.236153 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.235836 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="42a77847-5a9e-4825-80db-5f0b19133f62" containerName="payload-processing" Oct 30 18:07:35.236153 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.235842 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddb82e7c-5038-46b7-937e-c9ae4b12b2c0" containerName="istio-proxy" Oct 30 18:07:35.240931 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.240909 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7" Oct 30 18:07:35.259756 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.259725 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7"] Oct 30 18:07:35.267729 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.267693 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-95f55d\"/\"kube-root-ca.crt\"" Oct 30 18:07:35.270325 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.270290 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-95f55d\"/\"openshift-service-ca.crt\"" Oct 30 18:07:35.270325 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.270290 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-95f55d\"/\"ipp-route-95f55d-kserve-self-signed-certs\"" Oct 30 18:07:35.270532 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.270376 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-95f55d\"/\"default-dockercfg-cm45f\"" Oct 30 18:07:35.347065 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.347024 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgh29\" (UniqueName: \"kubernetes.io/projected/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-kube-api-access-sgh29\") pod \"ipp-route-95f55d-kserve-64f7586f5c-fh9x7\" (UID: \"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7\") " pod="ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7" Oct 30 18:07:35.347065 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.347069 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-tls-certs\") pod \"ipp-route-95f55d-kserve-64f7586f5c-fh9x7\" (UID: \"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7\") " pod="ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7" Oct 30 18:07:35.347303 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.347158 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-tmp-dir\") pod \"ipp-route-95f55d-kserve-64f7586f5c-fh9x7\" (UID: \"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7\") " pod="ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7" Oct 30 18:07:35.347303 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.347233 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-home\") pod \"ipp-route-95f55d-kserve-64f7586f5c-fh9x7\" (UID: \"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7\") " pod="ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7" Oct 30 18:07:35.347303 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.347278 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-dshm\") pod \"ipp-route-95f55d-kserve-64f7586f5c-fh9x7\" (UID: \"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7\") " pod="ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7" Oct 30 18:07:35.347303 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.347301 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-model-cache\") pod \"ipp-route-95f55d-kserve-64f7586f5c-fh9x7\" (UID: \"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7\") " pod="ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7" Oct 30 18:07:35.448174 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.448145 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgh29\" (UniqueName: \"kubernetes.io/projected/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-kube-api-access-sgh29\") pod \"ipp-route-95f55d-kserve-64f7586f5c-fh9x7\" (UID: \"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7\") " pod="ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7" Oct 30 18:07:35.448174 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.448179 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-tls-certs\") pod \"ipp-route-95f55d-kserve-64f7586f5c-fh9x7\" (UID: \"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7\") " pod="ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7" Oct 30 18:07:35.448454 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.448203 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-tmp-dir\") pod \"ipp-route-95f55d-kserve-64f7586f5c-fh9x7\" (UID: \"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7\") " pod="ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7" Oct 30 18:07:35.448454 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.448229 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-home\") pod \"ipp-route-95f55d-kserve-64f7586f5c-fh9x7\" (UID: \"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7\") " pod="ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7" Oct 30 18:07:35.448454 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.448249 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-dshm\") pod \"ipp-route-95f55d-kserve-64f7586f5c-fh9x7\" (UID: \"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7\") " pod="ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7" Oct 30 18:07:35.448454 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.448265 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-model-cache\") pod \"ipp-route-95f55d-kserve-64f7586f5c-fh9x7\" (UID: \"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7\") " pod="ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7" Oct 30 18:07:35.448683 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.448661 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-home\") pod \"ipp-route-95f55d-kserve-64f7586f5c-fh9x7\" (UID: \"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7\") " pod="ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7" Oct 30 18:07:35.448752 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.448728 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-tmp-dir\") pod \"ipp-route-95f55d-kserve-64f7586f5c-fh9x7\" (UID: \"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7\") " pod="ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7" Oct 30 18:07:35.448793 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.448757 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-model-cache\") pod \"ipp-route-95f55d-kserve-64f7586f5c-fh9x7\" (UID: \"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7\") " pod="ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7" Oct 30 18:07:35.450583 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.450562 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-dshm\") pod \"ipp-route-95f55d-kserve-64f7586f5c-fh9x7\" (UID: \"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7\") " pod="ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7" Oct 30 18:07:35.450749 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.450732 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-tls-certs\") pod \"ipp-route-95f55d-kserve-64f7586f5c-fh9x7\" (UID: \"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7\") " pod="ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7" Oct 30 18:07:35.471466 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.471441 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgh29\" (UniqueName: \"kubernetes.io/projected/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-kube-api-access-sgh29\") pod \"ipp-route-95f55d-kserve-64f7586f5c-fh9x7\" (UID: \"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7\") " pod="ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7" Oct 30 18:07:35.560337 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.560254 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7" Oct 30 18:07:35.865603 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:35.865579 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7"] Oct 30 18:07:35.867021 ip-10-0-139-222 kubenswrapper[4965]: W1030 18:07:35.866992 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e8b2094_be3e_4a33_92a5_7e2ca6e0b8c7.slice/crio-dbf52d828191147d272272ed911795c7cc0ee319b2a723adbe7a744a1955f642 WatchSource:0}: Error finding container dbf52d828191147d272272ed911795c7cc0ee319b2a723adbe7a744a1955f642: Status 404 returned error can't find the container with id dbf52d828191147d272272ed911795c7cc0ee319b2a723adbe7a744a1955f642 Oct 30 18:07:35.998791 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:07:35.998757 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:07:36.780768 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:36.780727 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7" event={"ID":"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7","Type":"ContainerStarted","Data":"d3eb557bc7658c75873df26175210dacc8c5c26a12ee53ecda5f6ef17478bc26"} Oct 30 18:07:36.780768 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:36.780773 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7" event={"ID":"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7","Type":"ContainerStarted","Data":"dbf52d828191147d272272ed911795c7cc0ee319b2a723adbe7a744a1955f642"} Oct 30 18:07:36.824550 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:36.824496 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7" podStartSLOduration=1.824481956 podStartE2EDuration="1.824481956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:36.822199153 +0000 UTC m=+3314.874299087" watchObservedRunningTime="2026-10-30 18:07:36.824481956 +0000 UTC m=+3314.876581895" Oct 30 18:07:37.561122 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:37.561084 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7" Oct 30 18:07:37.625432 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:37.625405 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:07:38.298792 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.298766 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 18:07:38.381651 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.381617 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4d5fb88-e105-459f-a9cd-959eb131c883-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d5fb88-e105-459f-a9cd-959eb131c883-istio-podinfo\") pod \"d4d5fb88-e105-459f-a9cd-959eb131c883\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " Oct 30 18:07:38.382003 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.381658 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-workload-certs\") pod \"d4d5fb88-e105-459f-a9cd-959eb131c883\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " Oct 30 18:07:38.382003 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.381684 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-istio-envoy\") pod \"d4d5fb88-e105-459f-a9cd-959eb131c883\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " Oct 30 18:07:38.382003 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.381717 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-istio-data\") pod \"d4d5fb88-e105-459f-a9cd-959eb131c883\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " Oct 30 18:07:38.382003 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.381760 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4d5fb88-e105-459f-a9cd-959eb131c883-istio-token\" (UniqueName: \"kubernetes.io/projected/d4d5fb88-e105-459f-a9cd-959eb131c883-istio-token\") pod \"d4d5fb88-e105-459f-a9cd-959eb131c883\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " Oct 30 18:07:38.382003 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.381811 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4d5fb88-e105-459f-a9cd-959eb131c883-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d5fb88-e105-459f-a9cd-959eb131c883-istiod-ca-cert\") pod \"d4d5fb88-e105-459f-a9cd-959eb131c883\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " Oct 30 18:07:38.382003 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.381906 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4d5fb88-e105-459f-a9cd-959eb131c883-kube-api-access-pkxlr\" (UniqueName: \"kubernetes.io/projected/d4d5fb88-e105-459f-a9cd-959eb131c883-kube-api-access-pkxlr\") pod \"d4d5fb88-e105-459f-a9cd-959eb131c883\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " Oct 30 18:07:38.382003 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.381939 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-credential-socket\") pod \"d4d5fb88-e105-459f-a9cd-959eb131c883\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " Oct 30 18:07:38.382003 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.382006 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-workload-socket\") pod \"d4d5fb88-e105-459f-a9cd-959eb131c883\" (UID: \"d4d5fb88-e105-459f-a9cd-959eb131c883\") " Oct 30 18:07:38.382972 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.382029 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-workload-certs" pod "d4d5fb88-e105-459f-a9cd-959eb131c883" (UID: "d4d5fb88-e105-459f-a9cd-959eb131c883"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:07:38.382972 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.382213 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-istio-data" pod "d4d5fb88-e105-459f-a9cd-959eb131c883" (UID: "d4d5fb88-e105-459f-a9cd-959eb131c883"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:07:38.382972 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.382325 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4d5fb88-e105-459f-a9cd-959eb131c883-istiod-ca-cert" pod "d4d5fb88-e105-459f-a9cd-959eb131c883" (UID: "d4d5fb88-e105-459f-a9cd-959eb131c883"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:07:38.382972 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.382422 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-workload-socket" pod "d4d5fb88-e105-459f-a9cd-959eb131c883" (UID: "d4d5fb88-e105-459f-a9cd-959eb131c883"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:07:38.382972 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.382580 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:38.382972 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.382585 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-credential-socket" pod "d4d5fb88-e105-459f-a9cd-959eb131c883" (UID: "d4d5fb88-e105-459f-a9cd-959eb131c883"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:07:38.382972 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.382599 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:38.382972 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.382611 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d5fb88-e105-459f-a9cd-959eb131c883-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:38.382972 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.382620 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:38.384616 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.384574 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-istio-envoy" pod "d4d5fb88-e105-459f-a9cd-959eb131c883" (UID: "d4d5fb88-e105-459f-a9cd-959eb131c883"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:07:38.384849 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.384814 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4d5fb88-e105-459f-a9cd-959eb131c883-kube-api-access-pkxlr" pod "d4d5fb88-e105-459f-a9cd-959eb131c883" (UID: "d4d5fb88-e105-459f-a9cd-959eb131c883"). InnerVolumeSpecName "kube-api-access-pkxlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:07:38.385155 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.385124 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4d5fb88-e105-459f-a9cd-959eb131c883-istio-token" pod "d4d5fb88-e105-459f-a9cd-959eb131c883" (UID: "d4d5fb88-e105-459f-a9cd-959eb131c883"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:07:38.385555 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.385535 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4d5fb88-e105-459f-a9cd-959eb131c883-istio-podinfo" pod "d4d5fb88-e105-459f-a9cd-959eb131c883" (UID: "d4d5fb88-e105-459f-a9cd-959eb131c883"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:07:38.483963 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.483929 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d5fb88-e105-459f-a9cd-959eb131c883-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:38.483963 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.483959 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:38.483963 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.483967 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4d5fb88-e105-459f-a9cd-959eb131c883-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:38.484213 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.483977 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkxlr\" (UniqueName: \"kubernetes.io/projected/d4d5fb88-e105-459f-a9cd-959eb131c883-kube-api-access-pkxlr\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:38.484213 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.483986 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d5fb88-e105-459f-a9cd-959eb131c883-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:07:38.793102 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.793016 4965 generic.go:356] "Generic (PLEG): container finished" podID="d4d5fb88-e105-459f-a9cd-959eb131c883" containerID="3dc003052fc409784cd6968b3a1625432af4248d4b15dd368a7618d27e3adda8" exitCode=0 Oct 30 18:07:38.793102 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.793071 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" Oct 30 18:07:38.793310 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.793098 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" event={"ID":"d4d5fb88-e105-459f-a9cd-959eb131c883","Type":"ContainerDied","Data":"3dc003052fc409784cd6968b3a1625432af4248d4b15dd368a7618d27e3adda8"} Oct 30 18:07:38.793310 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.793154 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz" event={"ID":"d4d5fb88-e105-459f-a9cd-959eb131c883","Type":"ContainerDied","Data":"913ede38f56b5c324b582fb16ffa5fb1c0be9701c8db080e4f70b3da8fd2cf07"} Oct 30 18:07:38.793310 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.793178 4965 scope.go:122] "RemoveContainer" containerID="3dc003052fc409784cd6968b3a1625432af4248d4b15dd368a7618d27e3adda8" Oct 30 18:07:38.803288 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.803269 4965 scope.go:122] "RemoveContainer" containerID="3dc003052fc409784cd6968b3a1625432af4248d4b15dd368a7618d27e3adda8" Oct 30 18:07:38.803550 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:07:38.803524 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dc003052fc409784cd6968b3a1625432af4248d4b15dd368a7618d27e3adda8\": container with ID starting with 3dc003052fc409784cd6968b3a1625432af4248d4b15dd368a7618d27e3adda8 not found: ID does not exist" containerID="3dc003052fc409784cd6968b3a1625432af4248d4b15dd368a7618d27e3adda8" Oct 30 18:07:38.803602 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.803564 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dc003052fc409784cd6968b3a1625432af4248d4b15dd368a7618d27e3adda8"} err="failed to get container status \"3dc003052fc409784cd6968b3a1625432af4248d4b15dd368a7618d27e3adda8\": rpc error: code = NotFound desc = could not find container \"3dc003052fc409784cd6968b3a1625432af4248d4b15dd368a7618d27e3adda8\": container with ID starting with 3dc003052fc409784cd6968b3a1625432af4248d4b15dd368a7618d27e3adda8 not found: ID does not exist" Oct 30 18:07:38.857487 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.857451 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz"] Oct 30 18:07:38.869714 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:38.869680 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2p4qz"] Oct 30 18:07:40.098287 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:40.098257 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4d5fb88-e105-459f-a9cd-959eb131c883" path="/var/lib/kubelet/pods/d4d5fb88-e105-459f-a9cd-959eb131c883/volumes" Oct 30 18:07:45.310150 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:45.310113 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-c31c73-7c486cc84-tggjn"] Oct 30 18:07:45.339442 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:45.339412 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c31c73-b54b865dc-568gf"] Oct 30 18:07:45.560530 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:45.560446 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7" Oct 30 18:07:45.579351 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:45.579321 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7" Oct 30 18:07:45.846767 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:45.846734 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7" Oct 30 18:07:50.311051 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:50.310998 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c31c73-7c486cc84-tggjn" podUID="7357d692-af47-49e1-a236-dd584604557e" containerName="payload-processing" containerID="cri-o://6e297a1d7e19a64de4b45a043bc2fd9d37812546e9a66b28099ebe21df9197ec" gracePeriod=25 Oct 30 18:07:50.340335 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:50.340284 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c31c73-b54b865dc-568gf" podUID="1bd96d50-e3b8-419c-9d18-20b88a540412" containerName="payload-pre-processing" containerID="cri-o://51663491535a5f6ab3ec3aedbcd4fa0607c2594b12d5ad2092057346761a3f77" gracePeriod=25 Oct 30 18:07:55.422482 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:55.422440 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c31c73-b54b865dc-568gf" podUID="1bd96d50-e3b8-419c-9d18-20b88a540412" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:07:55.422922 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:55.422480 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c31c73-7c486cc84-tggjn" podUID="7357d692-af47-49e1-a236-dd584604557e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:07:55.609490 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:55.609451 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:07:59.560077 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:59.560044 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2"] Oct 30 18:07:59.560575 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:59.560269 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" podUID="6497147f-bad2-434c-a6e4-45b247b75fc2" containerName="istio-proxy" containerID="cri-o://5e32b8689c8d76a9b29646ab656bc54e8c6b8bbbfa5f1d748ed5f0bbca51e53e" gracePeriod=30 Oct 30 18:07:59.847259 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:59.847217 4965 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:07:59.847438 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:07:59.847281 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" podUID="6497147f-bad2-434c-a6e4-45b247b75fc2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:08:04.823814 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.823789 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 18:08:04.885061 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.884994 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6497147f-bad2-434c-a6e4-45b247b75fc2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6497147f-bad2-434c-a6e4-45b247b75fc2-istio-podinfo\") pod \"6497147f-bad2-434c-a6e4-45b247b75fc2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " Oct 30 18:08:04.885061 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.885044 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-workload-socket\") pod \"6497147f-bad2-434c-a6e4-45b247b75fc2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " Oct 30 18:08:04.885300 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.885080 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-istio-envoy\") pod \"6497147f-bad2-434c-a6e4-45b247b75fc2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " Oct 30 18:08:04.885300 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.885115 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6497147f-bad2-434c-a6e4-45b247b75fc2-istio-token\" (UniqueName: \"kubernetes.io/projected/6497147f-bad2-434c-a6e4-45b247b75fc2-istio-token\") pod \"6497147f-bad2-434c-a6e4-45b247b75fc2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " Oct 30 18:08:04.885300 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.885155 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-istio-data\") pod \"6497147f-bad2-434c-a6e4-45b247b75fc2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " Oct 30 18:08:04.885300 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.885202 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6497147f-bad2-434c-a6e4-45b247b75fc2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6497147f-bad2-434c-a6e4-45b247b75fc2-istiod-ca-cert\") pod \"6497147f-bad2-434c-a6e4-45b247b75fc2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " Oct 30 18:08:04.885300 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.885252 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-workload-certs\") pod \"6497147f-bad2-434c-a6e4-45b247b75fc2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " Oct 30 18:08:04.885589 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.885351 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6497147f-bad2-434c-a6e4-45b247b75fc2-kube-api-access-k698f\" (UniqueName: \"kubernetes.io/projected/6497147f-bad2-434c-a6e4-45b247b75fc2-kube-api-access-k698f\") pod \"6497147f-bad2-434c-a6e4-45b247b75fc2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " Oct 30 18:08:04.885589 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.885394 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-credential-socket\") pod \"6497147f-bad2-434c-a6e4-45b247b75fc2\" (UID: \"6497147f-bad2-434c-a6e4-45b247b75fc2\") " Oct 30 18:08:04.885589 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.885399 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-workload-socket" pod "6497147f-bad2-434c-a6e4-45b247b75fc2" (UID: "6497147f-bad2-434c-a6e4-45b247b75fc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:04.885763 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.885605 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-istio-data" pod "6497147f-bad2-434c-a6e4-45b247b75fc2" (UID: "6497147f-bad2-434c-a6e4-45b247b75fc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:04.885763 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.885620 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-credential-socket" pod "6497147f-bad2-434c-a6e4-45b247b75fc2" (UID: "6497147f-bad2-434c-a6e4-45b247b75fc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:04.885763 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.885707 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6497147f-bad2-434c-a6e4-45b247b75fc2-istiod-ca-cert" pod "6497147f-bad2-434c-a6e4-45b247b75fc2" (UID: "6497147f-bad2-434c-a6e4-45b247b75fc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:08:04.885965 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.885836 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-workload-certs" pod "6497147f-bad2-434c-a6e4-45b247b75fc2" (UID: "6497147f-bad2-434c-a6e4-45b247b75fc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:04.886404 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.886305 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6497147f-bad2-434c-a6e4-45b247b75fc2-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:04.886404 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.886334 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:04.886404 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.886347 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:04.886404 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.886362 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:04.886404 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.886375 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:04.887934 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.887901 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-istio-envoy" pod "6497147f-bad2-434c-a6e4-45b247b75fc2" (UID: "6497147f-bad2-434c-a6e4-45b247b75fc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:04.888132 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.888100 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6497147f-bad2-434c-a6e4-45b247b75fc2-kube-api-access-k698f" pod "6497147f-bad2-434c-a6e4-45b247b75fc2" (UID: "6497147f-bad2-434c-a6e4-45b247b75fc2"). InnerVolumeSpecName "kube-api-access-k698f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:08:04.888259 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.888143 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6497147f-bad2-434c-a6e4-45b247b75fc2-istio-podinfo" pod "6497147f-bad2-434c-a6e4-45b247b75fc2" (UID: "6497147f-bad2-434c-a6e4-45b247b75fc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:08:04.891024 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.890997 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6497147f-bad2-434c-a6e4-45b247b75fc2-istio-token" pod "6497147f-bad2-434c-a6e4-45b247b75fc2" (UID: "6497147f-bad2-434c-a6e4-45b247b75fc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:08:04.954254 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.954217 4965 generic.go:356] "Generic (PLEG): container finished" podID="6497147f-bad2-434c-a6e4-45b247b75fc2" containerID="5e32b8689c8d76a9b29646ab656bc54e8c6b8bbbfa5f1d748ed5f0bbca51e53e" exitCode=0 Oct 30 18:08:04.954423 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.954287 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" Oct 30 18:08:04.954423 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.954310 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" event={"ID":"6497147f-bad2-434c-a6e4-45b247b75fc2","Type":"ContainerDied","Data":"5e32b8689c8d76a9b29646ab656bc54e8c6b8bbbfa5f1d748ed5f0bbca51e53e"} Oct 30 18:08:04.954423 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.954375 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2" event={"ID":"6497147f-bad2-434c-a6e4-45b247b75fc2","Type":"ContainerDied","Data":"f806d53241173bc857f8435ee8bcce81d16e8b6c6fcb62ffcd9a4c411bed7202"} Oct 30 18:08:04.954423 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.954399 4965 scope.go:122] "RemoveContainer" containerID="5e32b8689c8d76a9b29646ab656bc54e8c6b8bbbfa5f1d748ed5f0bbca51e53e" Oct 30 18:08:04.965467 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.965450 4965 scope.go:122] "RemoveContainer" containerID="1ec22adeee1cac2f41e7428a64b322f23700d26c42777a02438683ab45754b0f" Oct 30 18:08:04.974148 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.974125 4965 scope.go:122] "RemoveContainer" containerID="5e32b8689c8d76a9b29646ab656bc54e8c6b8bbbfa5f1d748ed5f0bbca51e53e" Oct 30 18:08:04.974467 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:08:04.974442 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e32b8689c8d76a9b29646ab656bc54e8c6b8bbbfa5f1d748ed5f0bbca51e53e\": container with ID starting with 5e32b8689c8d76a9b29646ab656bc54e8c6b8bbbfa5f1d748ed5f0bbca51e53e not found: ID does not exist" containerID="5e32b8689c8d76a9b29646ab656bc54e8c6b8bbbfa5f1d748ed5f0bbca51e53e" Oct 30 18:08:04.974553 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.974477 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e32b8689c8d76a9b29646ab656bc54e8c6b8bbbfa5f1d748ed5f0bbca51e53e"} err="failed to get container status \"5e32b8689c8d76a9b29646ab656bc54e8c6b8bbbfa5f1d748ed5f0bbca51e53e\": rpc error: code = NotFound desc = could not find container \"5e32b8689c8d76a9b29646ab656bc54e8c6b8bbbfa5f1d748ed5f0bbca51e53e\": container with ID starting with 5e32b8689c8d76a9b29646ab656bc54e8c6b8bbbfa5f1d748ed5f0bbca51e53e not found: ID does not exist" Oct 30 18:08:04.974553 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.974497 4965 scope.go:122] "RemoveContainer" containerID="1ec22adeee1cac2f41e7428a64b322f23700d26c42777a02438683ab45754b0f" Oct 30 18:08:04.974767 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:08:04.974747 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ec22adeee1cac2f41e7428a64b322f23700d26c42777a02438683ab45754b0f\": container with ID starting with 1ec22adeee1cac2f41e7428a64b322f23700d26c42777a02438683ab45754b0f not found: ID does not exist" containerID="1ec22adeee1cac2f41e7428a64b322f23700d26c42777a02438683ab45754b0f" Oct 30 18:08:04.974811 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.974783 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ec22adeee1cac2f41e7428a64b322f23700d26c42777a02438683ab45754b0f"} err="failed to get container status \"1ec22adeee1cac2f41e7428a64b322f23700d26c42777a02438683ab45754b0f\": rpc error: code = NotFound desc = could not find container \"1ec22adeee1cac2f41e7428a64b322f23700d26c42777a02438683ab45754b0f\": container with ID starting with 1ec22adeee1cac2f41e7428a64b322f23700d26c42777a02438683ab45754b0f not found: ID does not exist" Oct 30 18:08:04.988105 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.988085 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k698f\" (UniqueName: \"kubernetes.io/projected/6497147f-bad2-434c-a6e4-45b247b75fc2-kube-api-access-k698f\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:04.988105 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.988107 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6497147f-bad2-434c-a6e4-45b247b75fc2-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:04.988243 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.988117 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6497147f-bad2-434c-a6e4-45b247b75fc2-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:04.988243 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:04.988125 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6497147f-bad2-434c-a6e4-45b247b75fc2-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:05.007660 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:05.007633 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2"] Oct 30 18:08:05.019607 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:05.019586 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-q8hg2"] Oct 30 18:08:05.421826 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:05.421790 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c31c73-7c486cc84-tggjn" podUID="7357d692-af47-49e1-a236-dd584604557e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:08:05.422040 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:05.421884 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c31c73-b54b865dc-568gf" podUID="1bd96d50-e3b8-419c-9d18-20b88a540412" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:08:05.996059 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:08:05.996017 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:08:06.098166 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:06.098137 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6497147f-bad2-434c-a6e4-45b247b75fc2" path="/var/lib/kubelet/pods/6497147f-bad2-434c-a6e4-45b247b75fc2/volumes" Oct 30 18:08:10.601628 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:10.601602 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c31c73-7c486cc84-tggjn" Oct 30 18:08:10.651086 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:10.651055 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c31c73-b54b865dc-568gf" Oct 30 18:08:10.749399 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:10.749361 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bd96d50-e3b8-419c-9d18-20b88a540412-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bd96d50-e3b8-419c-9d18-20b88a540412-plugins-config-volume\") pod \"1bd96d50-e3b8-419c-9d18-20b88a540412\" (UID: \"1bd96d50-e3b8-419c-9d18-20b88a540412\") " Oct 30 18:08:10.749586 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:10.749413 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7357d692-af47-49e1-a236-dd584604557e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7357d692-af47-49e1-a236-dd584604557e-plugins-config-volume\") pod \"7357d692-af47-49e1-a236-dd584604557e\" (UID: \"7357d692-af47-49e1-a236-dd584604557e\") " Oct 30 18:08:10.749586 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:10.749537 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7357d692-af47-49e1-a236-dd584604557e-kube-api-access-t9lkg\" (UniqueName: \"kubernetes.io/projected/7357d692-af47-49e1-a236-dd584604557e-kube-api-access-t9lkg\") pod \"7357d692-af47-49e1-a236-dd584604557e\" (UID: \"7357d692-af47-49e1-a236-dd584604557e\") " Oct 30 18:08:10.749709 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:10.749589 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bd96d50-e3b8-419c-9d18-20b88a540412-kube-api-access-lzwzh\" (UniqueName: \"kubernetes.io/projected/1bd96d50-e3b8-419c-9d18-20b88a540412-kube-api-access-lzwzh\") pod \"1bd96d50-e3b8-419c-9d18-20b88a540412\" (UID: \"1bd96d50-e3b8-419c-9d18-20b88a540412\") " Oct 30 18:08:10.749775 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:10.749745 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bd96d50-e3b8-419c-9d18-20b88a540412-plugins-config-volume" pod "1bd96d50-e3b8-419c-9d18-20b88a540412" (UID: "1bd96d50-e3b8-419c-9d18-20b88a540412"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:08:10.749827 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:10.749792 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7357d692-af47-49e1-a236-dd584604557e-plugins-config-volume" pod "7357d692-af47-49e1-a236-dd584604557e" (UID: "7357d692-af47-49e1-a236-dd584604557e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:08:10.750099 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:10.750076 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bd96d50-e3b8-419c-9d18-20b88a540412-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:10.750174 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:10.750103 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7357d692-af47-49e1-a236-dd584604557e-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:10.751799 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:10.751780 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bd96d50-e3b8-419c-9d18-20b88a540412-kube-api-access-lzwzh" pod "1bd96d50-e3b8-419c-9d18-20b88a540412" (UID: "1bd96d50-e3b8-419c-9d18-20b88a540412"). InnerVolumeSpecName "kube-api-access-lzwzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:08:10.751879 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:10.751809 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7357d692-af47-49e1-a236-dd584604557e-kube-api-access-t9lkg" pod "7357d692-af47-49e1-a236-dd584604557e" (UID: "7357d692-af47-49e1-a236-dd584604557e"). InnerVolumeSpecName "kube-api-access-t9lkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:08:10.851160 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:10.851121 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9lkg\" (UniqueName: \"kubernetes.io/projected/7357d692-af47-49e1-a236-dd584604557e-kube-api-access-t9lkg\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:10.851380 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:10.851356 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzwzh\" (UniqueName: \"kubernetes.io/projected/1bd96d50-e3b8-419c-9d18-20b88a540412-kube-api-access-lzwzh\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:10.989314 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:10.989279 4965 generic.go:356] "Generic (PLEG): container finished" podID="1bd96d50-e3b8-419c-9d18-20b88a540412" containerID="51663491535a5f6ab3ec3aedbcd4fa0607c2594b12d5ad2092057346761a3f77" exitCode=0 Oct 30 18:08:10.989512 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:10.989324 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c31c73-b54b865dc-568gf" Oct 30 18:08:10.989512 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:10.989367 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c31c73-b54b865dc-568gf" event={"ID":"1bd96d50-e3b8-419c-9d18-20b88a540412","Type":"ContainerDied","Data":"51663491535a5f6ab3ec3aedbcd4fa0607c2594b12d5ad2092057346761a3f77"} Oct 30 18:08:10.989512 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:10.989422 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c31c73-b54b865dc-568gf" event={"ID":"1bd96d50-e3b8-419c-9d18-20b88a540412","Type":"ContainerDied","Data":"79254178d24cdd9c5345e6a723056e4379309e731518676231a5bc5f8fc49954"} Oct 30 18:08:10.989512 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:10.989452 4965 scope.go:122] "RemoveContainer" containerID="51663491535a5f6ab3ec3aedbcd4fa0607c2594b12d5ad2092057346761a3f77" Oct 30 18:08:10.990769 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:10.990748 4965 generic.go:356] "Generic (PLEG): container finished" podID="7357d692-af47-49e1-a236-dd584604557e" containerID="6e297a1d7e19a64de4b45a043bc2fd9d37812546e9a66b28099ebe21df9197ec" exitCode=0 Oct 30 18:08:10.990912 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:10.990785 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c31c73-7c486cc84-tggjn" Oct 30 18:08:10.990912 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:10.990789 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c31c73-7c486cc84-tggjn" event={"ID":"7357d692-af47-49e1-a236-dd584604557e","Type":"ContainerDied","Data":"6e297a1d7e19a64de4b45a043bc2fd9d37812546e9a66b28099ebe21df9197ec"} Oct 30 18:08:10.991049 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:10.990912 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c31c73-7c486cc84-tggjn" event={"ID":"7357d692-af47-49e1-a236-dd584604557e","Type":"ContainerDied","Data":"41cb2559cea2372f97201d174075a23bf805a57b0243c59ca8add4a5b1a4cf3e"} Oct 30 18:08:11.000387 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:11.000363 4965 scope.go:122] "RemoveContainer" containerID="51663491535a5f6ab3ec3aedbcd4fa0607c2594b12d5ad2092057346761a3f77" Oct 30 18:08:11.000682 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:08:11.000663 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51663491535a5f6ab3ec3aedbcd4fa0607c2594b12d5ad2092057346761a3f77\": container with ID starting with 51663491535a5f6ab3ec3aedbcd4fa0607c2594b12d5ad2092057346761a3f77 not found: ID does not exist" containerID="51663491535a5f6ab3ec3aedbcd4fa0607c2594b12d5ad2092057346761a3f77" Oct 30 18:08:11.000758 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:11.000688 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51663491535a5f6ab3ec3aedbcd4fa0607c2594b12d5ad2092057346761a3f77"} err="failed to get container status \"51663491535a5f6ab3ec3aedbcd4fa0607c2594b12d5ad2092057346761a3f77\": rpc error: code = NotFound desc = could not find container \"51663491535a5f6ab3ec3aedbcd4fa0607c2594b12d5ad2092057346761a3f77\": container with ID starting with 51663491535a5f6ab3ec3aedbcd4fa0607c2594b12d5ad2092057346761a3f77 not found: ID does not exist" Oct 30 18:08:11.000758 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:11.000712 4965 scope.go:122] "RemoveContainer" containerID="6e297a1d7e19a64de4b45a043bc2fd9d37812546e9a66b28099ebe21df9197ec" Oct 30 18:08:11.009421 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:11.009399 4965 scope.go:122] "RemoveContainer" containerID="6e297a1d7e19a64de4b45a043bc2fd9d37812546e9a66b28099ebe21df9197ec" Oct 30 18:08:11.009672 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:08:11.009650 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e297a1d7e19a64de4b45a043bc2fd9d37812546e9a66b28099ebe21df9197ec\": container with ID starting with 6e297a1d7e19a64de4b45a043bc2fd9d37812546e9a66b28099ebe21df9197ec not found: ID does not exist" containerID="6e297a1d7e19a64de4b45a043bc2fd9d37812546e9a66b28099ebe21df9197ec" Oct 30 18:08:11.009730 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:11.009679 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e297a1d7e19a64de4b45a043bc2fd9d37812546e9a66b28099ebe21df9197ec"} err="failed to get container status \"6e297a1d7e19a64de4b45a043bc2fd9d37812546e9a66b28099ebe21df9197ec\": rpc error: code = NotFound desc = could not find container \"6e297a1d7e19a64de4b45a043bc2fd9d37812546e9a66b28099ebe21df9197ec\": container with ID starting with 6e297a1d7e19a64de4b45a043bc2fd9d37812546e9a66b28099ebe21df9197ec not found: ID does not exist" Oct 30 18:08:11.039888 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:11.039843 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-c31c73-7c486cc84-tggjn"] Oct 30 18:08:11.043669 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:11.043646 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-c31c73-7c486cc84-tggjn"] Oct 30 18:08:11.076986 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:11.076951 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c31c73-b54b865dc-568gf"] Oct 30 18:08:11.080988 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:11.080964 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c31c73-b54b865dc-568gf"] Oct 30 18:08:12.097199 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:12.097170 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bd96d50-e3b8-419c-9d18-20b88a540412" path="/var/lib/kubelet/pods/1bd96d50-e3b8-419c-9d18-20b88a540412/volumes" Oct 30 18:08:12.097601 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:12.097548 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7357d692-af47-49e1-a236-dd584604557e" path="/var/lib/kubelet/pods/7357d692-af47-49e1-a236-dd584604557e/volumes" Oct 30 18:08:18.401573 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:18.401534 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:08:20.661159 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:20.661122 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2"] Oct 30 18:08:20.661704 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:20.661333 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" podUID="551401b3-f7a0-4b52-afac-f68219c56970" containerName="istio-proxy" containerID="cri-o://751c9b025e13eda4f301b78135ab90577126992f0a72c5f5707b4d8ff7375b0b" gracePeriod=30 Oct 30 18:08:24.308927 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:24.308902 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-5tdrf_d383c0b3-9245-4397-b031-38295a768fba/payload-processing/0.log" Oct 30 18:08:24.560135 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:24.560054 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-5tdrf_d383c0b3-9245-4397-b031-38295a768fba/payload-processing/0.log" Oct 30 18:08:25.933554 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:25.933496 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 18:08:26.019383 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.019348 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-credential-socket\") pod \"551401b3-f7a0-4b52-afac-f68219c56970\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " Oct 30 18:08:26.019559 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.019472 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-workload-certs\") pod \"551401b3-f7a0-4b52-afac-f68219c56970\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " Oct 30 18:08:26.019559 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.019543 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551401b3-f7a0-4b52-afac-f68219c56970-kube-api-access-phzfg\" (UniqueName: \"kubernetes.io/projected/551401b3-f7a0-4b52-afac-f68219c56970-kube-api-access-phzfg\") pod \"551401b3-f7a0-4b52-afac-f68219c56970\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " Oct 30 18:08:26.019695 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.019597 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/551401b3-f7a0-4b52-afac-f68219c56970-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551401b3-f7a0-4b52-afac-f68219c56970-istiod-ca-cert\") pod \"551401b3-f7a0-4b52-afac-f68219c56970\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " Oct 30 18:08:26.019695 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.019614 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-credential-socket" pod "551401b3-f7a0-4b52-afac-f68219c56970" (UID: "551401b3-f7a0-4b52-afac-f68219c56970"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:26.019695 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.019657 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-workload-socket\") pod \"551401b3-f7a0-4b52-afac-f68219c56970\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " Oct 30 18:08:26.019695 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.019694 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-istio-envoy\") pod \"551401b3-f7a0-4b52-afac-f68219c56970\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " Oct 30 18:08:26.019950 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.019740 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-istio-data\") pod \"551401b3-f7a0-4b52-afac-f68219c56970\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " Oct 30 18:08:26.019950 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.019797 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/551401b3-f7a0-4b52-afac-f68219c56970-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551401b3-f7a0-4b52-afac-f68219c56970-istio-podinfo\") pod \"551401b3-f7a0-4b52-afac-f68219c56970\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " Oct 30 18:08:26.019950 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.019835 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551401b3-f7a0-4b52-afac-f68219c56970-istio-token\" (UniqueName: \"kubernetes.io/projected/551401b3-f7a0-4b52-afac-f68219c56970-istio-token\") pod \"551401b3-f7a0-4b52-afac-f68219c56970\" (UID: \"551401b3-f7a0-4b52-afac-f68219c56970\") " Oct 30 18:08:26.019950 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.019839 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-workload-certs" pod "551401b3-f7a0-4b52-afac-f68219c56970" (UID: "551401b3-f7a0-4b52-afac-f68219c56970"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:26.020177 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.020002 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/551401b3-f7a0-4b52-afac-f68219c56970-istiod-ca-cert" pod "551401b3-f7a0-4b52-afac-f68219c56970" (UID: "551401b3-f7a0-4b52-afac-f68219c56970"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:08:26.020177 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.020100 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-istio-data" pod "551401b3-f7a0-4b52-afac-f68219c56970" (UID: "551401b3-f7a0-4b52-afac-f68219c56970"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:26.020330 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.020310 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:26.020399 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.020335 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:26.020399 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.020348 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551401b3-f7a0-4b52-afac-f68219c56970-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:26.020399 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.020361 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:26.020559 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.020393 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-workload-socket" pod "551401b3-f7a0-4b52-afac-f68219c56970" (UID: "551401b3-f7a0-4b52-afac-f68219c56970"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:26.022329 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.022296 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-istio-envoy" pod "551401b3-f7a0-4b52-afac-f68219c56970" (UID: "551401b3-f7a0-4b52-afac-f68219c56970"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:26.022672 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.022639 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551401b3-f7a0-4b52-afac-f68219c56970-kube-api-access-phzfg" pod "551401b3-f7a0-4b52-afac-f68219c56970" (UID: "551401b3-f7a0-4b52-afac-f68219c56970"). InnerVolumeSpecName "kube-api-access-phzfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:08:26.022788 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.022689 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/551401b3-f7a0-4b52-afac-f68219c56970-istio-podinfo" pod "551401b3-f7a0-4b52-afac-f68219c56970" (UID: "551401b3-f7a0-4b52-afac-f68219c56970"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:08:26.022788 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.022700 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551401b3-f7a0-4b52-afac-f68219c56970-istio-token" pod "551401b3-f7a0-4b52-afac-f68219c56970" (UID: "551401b3-f7a0-4b52-afac-f68219c56970"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:08:26.084456 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.084420 4965 generic.go:356] "Generic (PLEG): container finished" podID="551401b3-f7a0-4b52-afac-f68219c56970" containerID="751c9b025e13eda4f301b78135ab90577126992f0a72c5f5707b4d8ff7375b0b" exitCode=0 Oct 30 18:08:26.084641 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.084478 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" Oct 30 18:08:26.084641 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.084506 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" event={"ID":"551401b3-f7a0-4b52-afac-f68219c56970","Type":"ContainerDied","Data":"751c9b025e13eda4f301b78135ab90577126992f0a72c5f5707b4d8ff7375b0b"} Oct 30 18:08:26.084641 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.084605 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2" event={"ID":"551401b3-f7a0-4b52-afac-f68219c56970","Type":"ContainerDied","Data":"3158f58d69dc036305bd6302c4a7f52ae7a652617911902306efa978ceddc394"} Oct 30 18:08:26.084641 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.084619 4965 scope.go:122] "RemoveContainer" containerID="751c9b025e13eda4f301b78135ab90577126992f0a72c5f5707b4d8ff7375b0b" Oct 30 18:08:26.094965 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.094948 4965 scope.go:122] "RemoveContainer" containerID="ff1c89d80bdcd2394bd9266df2d3fa4dc1d7f322448d97b16ae1a5b52e64c535" Oct 30 18:08:26.107825 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.107800 4965 scope.go:122] "RemoveContainer" containerID="751c9b025e13eda4f301b78135ab90577126992f0a72c5f5707b4d8ff7375b0b" Oct 30 18:08:26.108137 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:08:26.108109 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"751c9b025e13eda4f301b78135ab90577126992f0a72c5f5707b4d8ff7375b0b\": container with ID starting with 751c9b025e13eda4f301b78135ab90577126992f0a72c5f5707b4d8ff7375b0b not found: ID does not exist" containerID="751c9b025e13eda4f301b78135ab90577126992f0a72c5f5707b4d8ff7375b0b" Oct 30 18:08:26.108254 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.108143 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"751c9b025e13eda4f301b78135ab90577126992f0a72c5f5707b4d8ff7375b0b"} err="failed to get container status \"751c9b025e13eda4f301b78135ab90577126992f0a72c5f5707b4d8ff7375b0b\": rpc error: code = NotFound desc = could not find container \"751c9b025e13eda4f301b78135ab90577126992f0a72c5f5707b4d8ff7375b0b\": container with ID starting with 751c9b025e13eda4f301b78135ab90577126992f0a72c5f5707b4d8ff7375b0b not found: ID does not exist" Oct 30 18:08:26.108254 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.108161 4965 scope.go:122] "RemoveContainer" containerID="ff1c89d80bdcd2394bd9266df2d3fa4dc1d7f322448d97b16ae1a5b52e64c535" Oct 30 18:08:26.108445 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:08:26.108429 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff1c89d80bdcd2394bd9266df2d3fa4dc1d7f322448d97b16ae1a5b52e64c535\": container with ID starting with ff1c89d80bdcd2394bd9266df2d3fa4dc1d7f322448d97b16ae1a5b52e64c535 not found: ID does not exist" containerID="ff1c89d80bdcd2394bd9266df2d3fa4dc1d7f322448d97b16ae1a5b52e64c535" Oct 30 18:08:26.108494 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.108448 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff1c89d80bdcd2394bd9266df2d3fa4dc1d7f322448d97b16ae1a5b52e64c535"} err="failed to get container status \"ff1c89d80bdcd2394bd9266df2d3fa4dc1d7f322448d97b16ae1a5b52e64c535\": rpc error: code = NotFound desc = could not find container \"ff1c89d80bdcd2394bd9266df2d3fa4dc1d7f322448d97b16ae1a5b52e64c535\": container with ID starting with ff1c89d80bdcd2394bd9266df2d3fa4dc1d7f322448d97b16ae1a5b52e64c535 not found: ID does not exist" Oct 30 18:08:26.121766 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.121743 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:26.121766 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.121765 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551401b3-f7a0-4b52-afac-f68219c56970-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:26.121927 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.121775 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551401b3-f7a0-4b52-afac-f68219c56970-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:26.121927 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.121783 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551401b3-f7a0-4b52-afac-f68219c56970-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:26.121927 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.121792 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phzfg\" (UniqueName: \"kubernetes.io/projected/551401b3-f7a0-4b52-afac-f68219c56970-kube-api-access-phzfg\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:26.136355 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.136329 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2"] Oct 30 18:08:26.151575 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.151543 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-2b6k2"] Oct 30 18:08:26.156880 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.156833 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj"] Oct 30 18:08:26.159472 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:26.159448 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7"] Oct 30 18:08:28.097977 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:28.097942 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="551401b3-f7a0-4b52-afac-f68219c56970" path="/var/lib/kubelet/pods/551401b3-f7a0-4b52-afac-f68219c56970/volumes" Oct 30 18:08:31.833145 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:31.833107 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-27567a-7c6f9f9747-hhm7h"] Oct 30 18:08:35.996131 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:08:35.996052 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:08:36.833665 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:36.833617 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-27567a-7c6f9f9747-hhm7h" podUID="9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d" containerName="payload-pre-processing" containerID="cri-o://3ba69e2ce57bf5f1e104e01f625eb17e39014f25b3c2f7b5276f13d42a0a54a3" gracePeriod=25 Oct 30 18:08:41.177699 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.177639 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" containerID="cri-o://bcf371609a3f73ce984f0bd7cc308488882ace523cacea3213819145d261afa9" gracePeriod=45 Oct 30 18:08:41.179612 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.179581 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" containerID="cri-o://a372cb5a6e6913b84b9f45d65bc3b622f669dda8eddf35f35d2135d1cacc9ab9" gracePeriod=45 Oct 30 18:08:41.472103 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.472076 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 18:08:41.475210 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.475192 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 18:08:41.485454 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.485434 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a3bb727-21cb-470a-9de6-ce748fcdddb1-home\" (UniqueName: \"kubernetes.io/empty-dir/2a3bb727-21cb-470a-9de6-ce748fcdddb1-home\") pod \"2a3bb727-21cb-470a-9de6-ce748fcdddb1\" (UID: \"2a3bb727-21cb-470a-9de6-ce748fcdddb1\") " Oct 30 18:08:41.485548 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.485469 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a3bb727-21cb-470a-9de6-ce748fcdddb1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a3bb727-21cb-470a-9de6-ce748fcdddb1-dshm\") pod \"2a3bb727-21cb-470a-9de6-ce748fcdddb1\" (UID: \"2a3bb727-21cb-470a-9de6-ce748fcdddb1\") " Oct 30 18:08:41.485548 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.485544 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a3bb727-21cb-470a-9de6-ce748fcdddb1-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3bb727-21cb-470a-9de6-ce748fcdddb1-tls-certs\") pod \"2a3bb727-21cb-470a-9de6-ce748fcdddb1\" (UID: \"2a3bb727-21cb-470a-9de6-ce748fcdddb1\") " Oct 30 18:08:41.485626 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.485580 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a3bb727-21cb-470a-9de6-ce748fcdddb1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a3bb727-21cb-470a-9de6-ce748fcdddb1-model-cache\") pod \"2a3bb727-21cb-470a-9de6-ce748fcdddb1\" (UID: \"2a3bb727-21cb-470a-9de6-ce748fcdddb1\") " Oct 30 18:08:41.485626 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.485599 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a3bb727-21cb-470a-9de6-ce748fcdddb1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a3bb727-21cb-470a-9de6-ce748fcdddb1-tmp-dir\") pod \"2a3bb727-21cb-470a-9de6-ce748fcdddb1\" (UID: \"2a3bb727-21cb-470a-9de6-ce748fcdddb1\") " Oct 30 18:08:41.485709 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.485636 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a3bb727-21cb-470a-9de6-ce748fcdddb1-kube-api-access-bgz5v\" (UniqueName: \"kubernetes.io/projected/2a3bb727-21cb-470a-9de6-ce748fcdddb1-kube-api-access-bgz5v\") pod \"2a3bb727-21cb-470a-9de6-ce748fcdddb1\" (UID: \"2a3bb727-21cb-470a-9de6-ce748fcdddb1\") " Oct 30 18:08:41.485761 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.485701 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3bb727-21cb-470a-9de6-ce748fcdddb1-home" pod "2a3bb727-21cb-470a-9de6-ce748fcdddb1" (UID: "2a3bb727-21cb-470a-9de6-ce748fcdddb1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:41.485909 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.485884 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3bb727-21cb-470a-9de6-ce748fcdddb1-model-cache" pod "2a3bb727-21cb-470a-9de6-ce748fcdddb1" (UID: "2a3bb727-21cb-470a-9de6-ce748fcdddb1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:41.485995 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.485899 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3bb727-21cb-470a-9de6-ce748fcdddb1-tmp-dir" pod "2a3bb727-21cb-470a-9de6-ce748fcdddb1" (UID: "2a3bb727-21cb-470a-9de6-ce748fcdddb1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:41.485995 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.485971 4965 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a3bb727-21cb-470a-9de6-ce748fcdddb1-home\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:41.485995 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.485983 4965 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a3bb727-21cb-470a-9de6-ce748fcdddb1-model-cache\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:41.486135 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.486012 4965 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a3bb727-21cb-470a-9de6-ce748fcdddb1-tmp-dir\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:41.487891 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.487840 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3bb727-21cb-470a-9de6-ce748fcdddb1-dshm" pod "2a3bb727-21cb-470a-9de6-ce748fcdddb1" (UID: "2a3bb727-21cb-470a-9de6-ce748fcdddb1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:41.487994 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.487917 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a3bb727-21cb-470a-9de6-ce748fcdddb1-kube-api-access-bgz5v" pod "2a3bb727-21cb-470a-9de6-ce748fcdddb1" (UID: "2a3bb727-21cb-470a-9de6-ce748fcdddb1"). InnerVolumeSpecName "kube-api-access-bgz5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:08:41.487994 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.487944 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a3bb727-21cb-470a-9de6-ce748fcdddb1-tls-certs" pod "2a3bb727-21cb-470a-9de6-ce748fcdddb1" (UID: "2a3bb727-21cb-470a-9de6-ce748fcdddb1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:08:41.586655 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.586616 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed4f3732-f99f-4218-a257-f66210947745-kube-api-access-krjhb\" (UniqueName: \"kubernetes.io/projected/ed4f3732-f99f-4218-a257-f66210947745-kube-api-access-krjhb\") pod \"ed4f3732-f99f-4218-a257-f66210947745\" (UID: \"ed4f3732-f99f-4218-a257-f66210947745\") " Oct 30 18:08:41.586857 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.586698 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed4f3732-f99f-4218-a257-f66210947745-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed4f3732-f99f-4218-a257-f66210947745-tmp-dir\") pod \"ed4f3732-f99f-4218-a257-f66210947745\" (UID: \"ed4f3732-f99f-4218-a257-f66210947745\") " Oct 30 18:08:41.586958 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.586852 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed4f3732-f99f-4218-a257-f66210947745-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed4f3732-f99f-4218-a257-f66210947745-tls-certs\") pod \"ed4f3732-f99f-4218-a257-f66210947745\" (UID: \"ed4f3732-f99f-4218-a257-f66210947745\") " Oct 30 18:08:41.586958 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.586920 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed4f3732-f99f-4218-a257-f66210947745-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed4f3732-f99f-4218-a257-f66210947745-dshm\") pod \"ed4f3732-f99f-4218-a257-f66210947745\" (UID: \"ed4f3732-f99f-4218-a257-f66210947745\") " Oct 30 18:08:41.587084 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.586984 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed4f3732-f99f-4218-a257-f66210947745-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed4f3732-f99f-4218-a257-f66210947745-model-cache\") pod \"ed4f3732-f99f-4218-a257-f66210947745\" (UID: \"ed4f3732-f99f-4218-a257-f66210947745\") " Oct 30 18:08:41.587084 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.587022 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed4f3732-f99f-4218-a257-f66210947745-home\" (UniqueName: \"kubernetes.io/empty-dir/ed4f3732-f99f-4218-a257-f66210947745-home\") pod \"ed4f3732-f99f-4218-a257-f66210947745\" (UID: \"ed4f3732-f99f-4218-a257-f66210947745\") " Oct 30 18:08:41.587477 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.587015 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed4f3732-f99f-4218-a257-f66210947745-tmp-dir" pod "ed4f3732-f99f-4218-a257-f66210947745" (UID: "ed4f3732-f99f-4218-a257-f66210947745"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:41.587642 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.587469 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed4f3732-f99f-4218-a257-f66210947745-home" pod "ed4f3732-f99f-4218-a257-f66210947745" (UID: "ed4f3732-f99f-4218-a257-f66210947745"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:41.587763 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.587690 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed4f3732-f99f-4218-a257-f66210947745-model-cache" pod "ed4f3732-f99f-4218-a257-f66210947745" (UID: "ed4f3732-f99f-4218-a257-f66210947745"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:41.587900 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.587483 4965 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3bb727-21cb-470a-9de6-ce748fcdddb1-tls-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:41.588025 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.587910 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgz5v\" (UniqueName: \"kubernetes.io/projected/2a3bb727-21cb-470a-9de6-ce748fcdddb1-kube-api-access-bgz5v\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:41.588025 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.587927 4965 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a3bb727-21cb-470a-9de6-ce748fcdddb1-dshm\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:41.589199 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.589173 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed4f3732-f99f-4218-a257-f66210947745-dshm" pod "ed4f3732-f99f-4218-a257-f66210947745" (UID: "ed4f3732-f99f-4218-a257-f66210947745"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:41.590049 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.590028 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed4f3732-f99f-4218-a257-f66210947745-kube-api-access-krjhb" pod "ed4f3732-f99f-4218-a257-f66210947745" (UID: "ed4f3732-f99f-4218-a257-f66210947745"). InnerVolumeSpecName "kube-api-access-krjhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:08:41.590687 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.590669 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed4f3732-f99f-4218-a257-f66210947745-tls-certs" pod "ed4f3732-f99f-4218-a257-f66210947745" (UID: "ed4f3732-f99f-4218-a257-f66210947745"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:08:41.689475 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.689373 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krjhb\" (UniqueName: \"kubernetes.io/projected/ed4f3732-f99f-4218-a257-f66210947745-kube-api-access-krjhb\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:41.689475 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.689422 4965 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed4f3732-f99f-4218-a257-f66210947745-tmp-dir\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:41.689475 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.689435 4965 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed4f3732-f99f-4218-a257-f66210947745-tls-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:41.689475 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.689442 4965 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed4f3732-f99f-4218-a257-f66210947745-dshm\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:41.689475 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.689450 4965 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed4f3732-f99f-4218-a257-f66210947745-model-cache\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:41.689475 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.689458 4965 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed4f3732-f99f-4218-a257-f66210947745-home\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:41.761666 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.761629 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c6d3a2c-wkpzc"] Oct 30 18:08:41.762161 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.762146 4965 cpu_manager.go:414] "containerMap: removing container" podUID="551401b3-f7a0-4b52-afac-f68219c56970" containerName="istio-proxy" Oct 30 18:08:41.762220 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.762163 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551401b3-f7a0-4b52-afac-f68219c56970" containerName="istio-proxy" Oct 30 18:08:41.762220 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.762173 4965 cpu_manager.go:414] "containerMap: removing container" podUID="551401b3-f7a0-4b52-afac-f68219c56970" containerName="istio-proxy" Oct 30 18:08:41.762220 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.762179 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551401b3-f7a0-4b52-afac-f68219c56970" containerName="istio-proxy" Oct 30 18:08:41.762220 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.762189 4965 cpu_manager.go:414] "containerMap: removing container" podUID="6497147f-bad2-434c-a6e4-45b247b75fc2" containerName="istio-proxy" Oct 30 18:08:41.762220 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.762195 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6497147f-bad2-434c-a6e4-45b247b75fc2" containerName="istio-proxy" Oct 30 18:08:41.762220 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.762204 4965 cpu_manager.go:414] "containerMap: removing container" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" Oct 30 18:08:41.762220 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.762210 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" Oct 30 18:08:41.762220 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.762223 4965 cpu_manager.go:414] "containerMap: removing container" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" Oct 30 18:08:41.762485 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.762227 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" Oct 30 18:08:41.762485 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.762234 4965 cpu_manager.go:414] "containerMap: removing container" podUID="7357d692-af47-49e1-a236-dd584604557e" containerName="payload-processing" Oct 30 18:08:41.762485 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.762248 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7357d692-af47-49e1-a236-dd584604557e" containerName="payload-processing" Oct 30 18:08:41.762485 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.762262 4965 cpu_manager.go:414] "containerMap: removing container" podUID="1bd96d50-e3b8-419c-9d18-20b88a540412" containerName="payload-pre-processing" Oct 30 18:08:41.762485 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.762267 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bd96d50-e3b8-419c-9d18-20b88a540412" containerName="payload-pre-processing" Oct 30 18:08:41.762485 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.762275 4965 cpu_manager.go:414] "containerMap: removing container" podUID="d4d5fb88-e105-459f-a9cd-959eb131c883" containerName="istio-proxy" Oct 30 18:08:41.762485 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.762280 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4d5fb88-e105-459f-a9cd-959eb131c883" containerName="istio-proxy" Oct 30 18:08:41.762485 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.762360 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bd96d50-e3b8-419c-9d18-20b88a540412" containerName="payload-pre-processing" Oct 30 18:08:41.762485 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.762369 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="6497147f-bad2-434c-a6e4-45b247b75fc2" containerName="istio-proxy" Oct 30 18:08:41.762485 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.762376 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4d5fb88-e105-459f-a9cd-959eb131c883" containerName="istio-proxy" Oct 30 18:08:41.762485 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.762386 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="551401b3-f7a0-4b52-afac-f68219c56970" containerName="istio-proxy" Oct 30 18:08:41.762485 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.762391 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="551401b3-f7a0-4b52-afac-f68219c56970" containerName="istio-proxy" Oct 30 18:08:41.762485 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.762398 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed4f3732-f99f-4218-a257-f66210947745" containerName="main" Oct 30 18:08:41.762485 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.762405 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerName="main" Oct 30 18:08:41.762485 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.762409 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="7357d692-af47-49e1-a236-dd584604557e" containerName="payload-processing" Oct 30 18:08:41.766915 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.766900 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6d3a2c-wkpzc" Oct 30 18:08:41.781724 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.781694 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:08:41.781947 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.781928 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\"" Oct 30 18:08:41.783320 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.783295 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c6d3a2c-wkpzc"] Oct 30 18:08:41.790214 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.790191 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93b77541-e76a-4b77-99f7-68e1e5e2a624-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c6d3a2c-wkpzc\" (UID: \"93b77541-e76a-4b77-99f7-68e1e5e2a624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6d3a2c-wkpzc" Oct 30 18:08:41.790345 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.790324 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93b77541-e76a-4b77-99f7-68e1e5e2a624-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c6d3a2c-wkpzc\" (UID: \"93b77541-e76a-4b77-99f7-68e1e5e2a624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6d3a2c-wkpzc" Oct 30 18:08:41.790442 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.790430 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jsnt\" (UniqueName: \"kubernetes.io/projected/93b77541-e76a-4b77-99f7-68e1e5e2a624-kube-api-access-4jsnt\") pod \"maas-api-revoke-sub-4c6d3a2c-wkpzc\" (UID: \"93b77541-e76a-4b77-99f7-68e1e5e2a624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6d3a2c-wkpzc" Oct 30 18:08:41.891412 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.891374 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jsnt\" (UniqueName: \"kubernetes.io/projected/93b77541-e76a-4b77-99f7-68e1e5e2a624-kube-api-access-4jsnt\") pod \"maas-api-revoke-sub-4c6d3a2c-wkpzc\" (UID: \"93b77541-e76a-4b77-99f7-68e1e5e2a624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6d3a2c-wkpzc" Oct 30 18:08:41.891622 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.891426 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93b77541-e76a-4b77-99f7-68e1e5e2a624-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c6d3a2c-wkpzc\" (UID: \"93b77541-e76a-4b77-99f7-68e1e5e2a624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6d3a2c-wkpzc" Oct 30 18:08:41.891622 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.891494 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93b77541-e76a-4b77-99f7-68e1e5e2a624-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c6d3a2c-wkpzc\" (UID: \"93b77541-e76a-4b77-99f7-68e1e5e2a624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6d3a2c-wkpzc" Oct 30 18:08:41.892117 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.892091 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93b77541-e76a-4b77-99f7-68e1e5e2a624-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c6d3a2c-wkpzc\" (UID: \"93b77541-e76a-4b77-99f7-68e1e5e2a624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6d3a2c-wkpzc" Oct 30 18:08:41.893846 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.893823 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93b77541-e76a-4b77-99f7-68e1e5e2a624-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c6d3a2c-wkpzc\" (UID: \"93b77541-e76a-4b77-99f7-68e1e5e2a624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6d3a2c-wkpzc" Oct 30 18:08:41.914398 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:41.914370 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jsnt\" (UniqueName: \"kubernetes.io/projected/93b77541-e76a-4b77-99f7-68e1e5e2a624-kube-api-access-4jsnt\") pod \"maas-api-revoke-sub-4c6d3a2c-wkpzc\" (UID: \"93b77541-e76a-4b77-99f7-68e1e5e2a624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6d3a2c-wkpzc" Oct 30 18:08:42.020910 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:42.020811 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-27567a-7c6f9f9747-hhm7h" podUID="9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:08:42.080427 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:42.080390 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6d3a2c-wkpzc" Oct 30 18:08:42.170742 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:42.170714 4965 generic.go:356] "Generic (PLEG): container finished" podID="ed4f3732-f99f-4218-a257-f66210947745" containerID="a372cb5a6e6913b84b9f45d65bc3b622f669dda8eddf35f35d2135d1cacc9ab9" exitCode=0 Oct 30 18:08:42.170953 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:42.170778 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" Oct 30 18:08:42.170953 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:42.170796 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" event={"ID":"ed4f3732-f99f-4218-a257-f66210947745","Type":"ContainerDied","Data":"a372cb5a6e6913b84b9f45d65bc3b622f669dda8eddf35f35d2135d1cacc9ab9"} Oct 30 18:08:42.170953 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:42.170844 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7" event={"ID":"ed4f3732-f99f-4218-a257-f66210947745","Type":"ContainerDied","Data":"7fb089398415d3105c13d9f249295a655ec0496058e513cc7278be6ea165f33a"} Oct 30 18:08:42.170953 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:42.170891 4965 scope.go:122] "RemoveContainer" containerID="a372cb5a6e6913b84b9f45d65bc3b622f669dda8eddf35f35d2135d1cacc9ab9" Oct 30 18:08:42.172496 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:42.172473 4965 generic.go:356] "Generic (PLEG): container finished" podID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" containerID="bcf371609a3f73ce984f0bd7cc308488882ace523cacea3213819145d261afa9" exitCode=0 Oct 30 18:08:42.172634 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:42.172616 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" event={"ID":"2a3bb727-21cb-470a-9de6-ce748fcdddb1","Type":"ContainerDied","Data":"bcf371609a3f73ce984f0bd7cc308488882ace523cacea3213819145d261afa9"} Oct 30 18:08:42.172701 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:42.172651 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" event={"ID":"2a3bb727-21cb-470a-9de6-ce748fcdddb1","Type":"ContainerDied","Data":"39735219d29a507f240bf5b85b045b482716aedce310c376f91aeba8a9c4665f"} Oct 30 18:08:42.172765 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:42.172710 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj" Oct 30 18:08:42.183270 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:42.182607 4965 scope.go:122] "RemoveContainer" containerID="a372cb5a6e6913b84b9f45d65bc3b622f669dda8eddf35f35d2135d1cacc9ab9" Oct 30 18:08:42.183270 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:08:42.182932 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a372cb5a6e6913b84b9f45d65bc3b622f669dda8eddf35f35d2135d1cacc9ab9\": container with ID starting with a372cb5a6e6913b84b9f45d65bc3b622f669dda8eddf35f35d2135d1cacc9ab9 not found: ID does not exist" containerID="a372cb5a6e6913b84b9f45d65bc3b622f669dda8eddf35f35d2135d1cacc9ab9" Oct 30 18:08:42.183270 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:42.182964 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a372cb5a6e6913b84b9f45d65bc3b622f669dda8eddf35f35d2135d1cacc9ab9"} err="failed to get container status \"a372cb5a6e6913b84b9f45d65bc3b622f669dda8eddf35f35d2135d1cacc9ab9\": rpc error: code = NotFound desc = could not find container \"a372cb5a6e6913b84b9f45d65bc3b622f669dda8eddf35f35d2135d1cacc9ab9\": container with ID starting with a372cb5a6e6913b84b9f45d65bc3b622f669dda8eddf35f35d2135d1cacc9ab9 not found: ID does not exist" Oct 30 18:08:42.183270 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:42.182987 4965 scope.go:122] "RemoveContainer" containerID="bcf371609a3f73ce984f0bd7cc308488882ace523cacea3213819145d261afa9" Oct 30 18:08:42.207628 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:42.207607 4965 scope.go:122] "RemoveContainer" containerID="bcf371609a3f73ce984f0bd7cc308488882ace523cacea3213819145d261afa9" Oct 30 18:08:42.207959 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:08:42.207933 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcf371609a3f73ce984f0bd7cc308488882ace523cacea3213819145d261afa9\": container with ID starting with bcf371609a3f73ce984f0bd7cc308488882ace523cacea3213819145d261afa9 not found: ID does not exist" containerID="bcf371609a3f73ce984f0bd7cc308488882ace523cacea3213819145d261afa9" Oct 30 18:08:42.208064 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:42.207966 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcf371609a3f73ce984f0bd7cc308488882ace523cacea3213819145d261afa9"} err="failed to get container status \"bcf371609a3f73ce984f0bd7cc308488882ace523cacea3213819145d261afa9\": rpc error: code = NotFound desc = could not find container \"bcf371609a3f73ce984f0bd7cc308488882ace523cacea3213819145d261afa9\": container with ID starting with bcf371609a3f73ce984f0bd7cc308488882ace523cacea3213819145d261afa9 not found: ID does not exist" Oct 30 18:08:42.221805 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:42.221778 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj"] Oct 30 18:08:42.225565 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:42.225545 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-27567a/distinct-2-w0-27567a-kserve-674f7b75fd-vv7cj"] Oct 30 18:08:42.262336 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:42.262310 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7"] Oct 30 18:08:42.264312 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:42.264291 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-27567a/distinct-w0-27567a-kserve-766b7789cc-6cnd7"] Oct 30 18:08:42.391024 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:42.390993 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c6d3a2c-wkpzc"] Oct 30 18:08:42.396076 ip-10-0-139-222 kubenswrapper[4965]: W1030 18:08:42.396046 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b77541_e76a_4b77_99f7_68e1e5e2a624.slice/crio-55b0f5306901b3bbf10e548057cab188aa026c8b9321a10a2a0e59ae1732f48f WatchSource:0}: Error finding container 55b0f5306901b3bbf10e548057cab188aa026c8b9321a10a2a0e59ae1732f48f: Status 404 returned error can't find the container with id 55b0f5306901b3bbf10e548057cab188aa026c8b9321a10a2a0e59ae1732f48f Oct 30 18:08:43.179725 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:43.179687 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6d3a2c-wkpzc" event={"ID":"93b77541-e76a-4b77-99f7-68e1e5e2a624","Type":"ContainerStarted","Data":"0c41b809b9a6d3dd6b5f8b476e76d77514baba6a7a196aaafdc2a5edbea24728"} Oct 30 18:08:43.179905 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:43.179750 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6d3a2c-wkpzc" event={"ID":"93b77541-e76a-4b77-99f7-68e1e5e2a624","Type":"ContainerStarted","Data":"55b0f5306901b3bbf10e548057cab188aa026c8b9321a10a2a0e59ae1732f48f"} Oct 30 18:08:43.220627 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:43.220577 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6d3a2c-wkpzc" podStartSLOduration=2.220564015 podStartE2EDuration="2.220564015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:08:43.218499299 +0000 UTC m=+3381.270599229" watchObservedRunningTime="2026-10-30 18:08:43.220564015 +0000 UTC m=+3381.272663946" Oct 30 18:08:44.098311 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:44.098278 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a3bb727-21cb-470a-9de6-ce748fcdddb1" path="/var/lib/kubelet/pods/2a3bb727-21cb-470a-9de6-ce748fcdddb1/volumes" Oct 30 18:08:44.098704 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:44.098690 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed4f3732-f99f-4218-a257-f66210947745" path="/var/lib/kubelet/pods/ed4f3732-f99f-4218-a257-f66210947745/volumes" Oct 30 18:08:44.188694 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:44.188663 4965 generic.go:356] "Generic (PLEG): container finished" podID="93b77541-e76a-4b77-99f7-68e1e5e2a624" containerID="0c41b809b9a6d3dd6b5f8b476e76d77514baba6a7a196aaafdc2a5edbea24728" exitCode=0 Oct 30 18:08:44.188892 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:44.188747 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6d3a2c-wkpzc" event={"ID":"93b77541-e76a-4b77-99f7-68e1e5e2a624","Type":"ContainerDied","Data":"0c41b809b9a6d3dd6b5f8b476e76d77514baba6a7a196aaafdc2a5edbea24728"} Oct 30 18:08:45.340186 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:45.340161 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6d3a2c-wkpzc" Oct 30 18:08:45.429176 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:45.429141 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93b77541-e76a-4b77-99f7-68e1e5e2a624-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93b77541-e76a-4b77-99f7-68e1e5e2a624-maas-api-xks-ca\") pod \"93b77541-e76a-4b77-99f7-68e1e5e2a624\" (UID: \"93b77541-e76a-4b77-99f7-68e1e5e2a624\") " Oct 30 18:08:45.429372 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:45.429196 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93b77541-e76a-4b77-99f7-68e1e5e2a624-kube-api-access-4jsnt\" (UniqueName: \"kubernetes.io/projected/93b77541-e76a-4b77-99f7-68e1e5e2a624-kube-api-access-4jsnt\") pod \"93b77541-e76a-4b77-99f7-68e1e5e2a624\" (UID: \"93b77541-e76a-4b77-99f7-68e1e5e2a624\") " Oct 30 18:08:45.429372 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:45.429270 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93b77541-e76a-4b77-99f7-68e1e5e2a624-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93b77541-e76a-4b77-99f7-68e1e5e2a624-maas-api-openshift-ca\") pod \"93b77541-e76a-4b77-99f7-68e1e5e2a624\" (UID: \"93b77541-e76a-4b77-99f7-68e1e5e2a624\") " Oct 30 18:08:45.429635 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:45.429612 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93b77541-e76a-4b77-99f7-68e1e5e2a624-maas-api-openshift-ca" pod "93b77541-e76a-4b77-99f7-68e1e5e2a624" (UID: "93b77541-e76a-4b77-99f7-68e1e5e2a624"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:08:45.431412 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:45.431390 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93b77541-e76a-4b77-99f7-68e1e5e2a624-maas-api-xks-ca" pod "93b77541-e76a-4b77-99f7-68e1e5e2a624" (UID: "93b77541-e76a-4b77-99f7-68e1e5e2a624"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:08:45.431521 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:45.431415 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b77541-e76a-4b77-99f7-68e1e5e2a624-kube-api-access-4jsnt" pod "93b77541-e76a-4b77-99f7-68e1e5e2a624" (UID: "93b77541-e76a-4b77-99f7-68e1e5e2a624"). InnerVolumeSpecName "kube-api-access-4jsnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:08:45.530340 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:45.530272 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93b77541-e76a-4b77-99f7-68e1e5e2a624-maas-api-xks-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:45.530340 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:45.530298 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jsnt\" (UniqueName: \"kubernetes.io/projected/93b77541-e76a-4b77-99f7-68e1e5e2a624-kube-api-access-4jsnt\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:45.530340 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:45.530308 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93b77541-e76a-4b77-99f7-68e1e5e2a624-maas-api-openshift-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:46.201706 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:46.201674 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6d3a2c-wkpzc" Oct 30 18:08:46.201706 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:46.201690 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6d3a2c-wkpzc" event={"ID":"93b77541-e76a-4b77-99f7-68e1e5e2a624","Type":"ContainerDied","Data":"55b0f5306901b3bbf10e548057cab188aa026c8b9321a10a2a0e59ae1732f48f"} Oct 30 18:08:46.202012 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:46.201738 4965 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55b0f5306901b3bbf10e548057cab188aa026c8b9321a10a2a0e59ae1732f48f" Oct 30 18:08:52.021519 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:52.021482 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-27567a-7c6f9f9747-hhm7h" podUID="9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:08:53.204826 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:53.204783 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm"] Oct 30 18:08:53.205366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:53.205015 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm" podUID="d9665073-6722-4274-bc2d-d453b06bd6c0" containerName="maas-api" containerID="cri-o://80d516712461197c10961cf3a0b8d86ff024c5cb3740af694dd0e7f8d213b0c0" gracePeriod=30 Oct 30 18:08:53.469181 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:53.469125 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm" Oct 30 18:08:53.614827 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:53.614792 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9665073-6722-4274-bc2d-d453b06bd6c0-kube-api-access-4jn2x\" (UniqueName: \"kubernetes.io/projected/d9665073-6722-4274-bc2d-d453b06bd6c0-kube-api-access-4jn2x\") pod \"d9665073-6722-4274-bc2d-d453b06bd6c0\" (UID: \"d9665073-6722-4274-bc2d-d453b06bd6c0\") " Oct 30 18:08:53.615003 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:53.614857 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9665073-6722-4274-bc2d-d453b06bd6c0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9665073-6722-4274-bc2d-d453b06bd6c0-metrics-certs\") pod \"d9665073-6722-4274-bc2d-d453b06bd6c0\" (UID: \"d9665073-6722-4274-bc2d-d453b06bd6c0\") " Oct 30 18:08:53.615062 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:53.615027 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9665073-6722-4274-bc2d-d453b06bd6c0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9665073-6722-4274-bc2d-d453b06bd6c0-openshift-service-ca\") pod \"d9665073-6722-4274-bc2d-d453b06bd6c0\" (UID: \"d9665073-6722-4274-bc2d-d453b06bd6c0\") " Oct 30 18:08:53.615105 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:53.615057 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9665073-6722-4274-bc2d-d453b06bd6c0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9665073-6722-4274-bc2d-d453b06bd6c0-maas-api-tls\") pod \"d9665073-6722-4274-bc2d-d453b06bd6c0\" (UID: \"d9665073-6722-4274-bc2d-d453b06bd6c0\") " Oct 30 18:08:53.615451 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:53.615424 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9665073-6722-4274-bc2d-d453b06bd6c0-openshift-service-ca" pod "d9665073-6722-4274-bc2d-d453b06bd6c0" (UID: "d9665073-6722-4274-bc2d-d453b06bd6c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:08:53.616110 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:53.616088 4965 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9665073-6722-4274-bc2d-d453b06bd6c0-openshift-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:53.617209 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:53.617173 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9665073-6722-4274-bc2d-d453b06bd6c0-metrics-certs" pod "d9665073-6722-4274-bc2d-d453b06bd6c0" (UID: "d9665073-6722-4274-bc2d-d453b06bd6c0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:08:53.617465 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:53.617424 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9665073-6722-4274-bc2d-d453b06bd6c0-maas-api-tls" pod "d9665073-6722-4274-bc2d-d453b06bd6c0" (UID: "d9665073-6722-4274-bc2d-d453b06bd6c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:08:53.617465 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:53.617440 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9665073-6722-4274-bc2d-d453b06bd6c0-kube-api-access-4jn2x" pod "d9665073-6722-4274-bc2d-d453b06bd6c0" (UID: "d9665073-6722-4274-bc2d-d453b06bd6c0"). InnerVolumeSpecName "kube-api-access-4jn2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:08:53.717590 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:53.717556 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jn2x\" (UniqueName: \"kubernetes.io/projected/d9665073-6722-4274-bc2d-d453b06bd6c0-kube-api-access-4jn2x\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:53.717590 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:53.717588 4965 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9665073-6722-4274-bc2d-d453b06bd6c0-metrics-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:53.717590 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:53.717598 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9665073-6722-4274-bc2d-d453b06bd6c0-maas-api-tls\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:54.246632 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:54.246599 4965 generic.go:356] "Generic (PLEG): container finished" podID="d9665073-6722-4274-bc2d-d453b06bd6c0" containerID="80d516712461197c10961cf3a0b8d86ff024c5cb3740af694dd0e7f8d213b0c0" exitCode=0 Oct 30 18:08:54.247094 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:54.246643 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm" Oct 30 18:08:54.247094 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:54.246682 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm" event={"ID":"d9665073-6722-4274-bc2d-d453b06bd6c0","Type":"ContainerDied","Data":"80d516712461197c10961cf3a0b8d86ff024c5cb3740af694dd0e7f8d213b0c0"} Oct 30 18:08:54.247094 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:54.246738 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm" event={"ID":"d9665073-6722-4274-bc2d-d453b06bd6c0","Type":"ContainerDied","Data":"395323d4fb41dba57c1727e9a7e664eafbfdaec930f31328ae8c9cfba69dcc9c"} Oct 30 18:08:54.247094 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:54.246766 4965 scope.go:122] "RemoveContainer" containerID="80d516712461197c10961cf3a0b8d86ff024c5cb3740af694dd0e7f8d213b0c0" Oct 30 18:08:54.255906 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:54.255857 4965 scope.go:122] "RemoveContainer" containerID="80d516712461197c10961cf3a0b8d86ff024c5cb3740af694dd0e7f8d213b0c0" Oct 30 18:08:54.256177 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:08:54.256158 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80d516712461197c10961cf3a0b8d86ff024c5cb3740af694dd0e7f8d213b0c0\": container with ID starting with 80d516712461197c10961cf3a0b8d86ff024c5cb3740af694dd0e7f8d213b0c0 not found: ID does not exist" containerID="80d516712461197c10961cf3a0b8d86ff024c5cb3740af694dd0e7f8d213b0c0" Oct 30 18:08:54.256234 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:54.256186 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80d516712461197c10961cf3a0b8d86ff024c5cb3740af694dd0e7f8d213b0c0"} err="failed to get container status \"80d516712461197c10961cf3a0b8d86ff024c5cb3740af694dd0e7f8d213b0c0\": rpc error: code = NotFound desc = could not find container \"80d516712461197c10961cf3a0b8d86ff024c5cb3740af694dd0e7f8d213b0c0\": container with ID starting with 80d516712461197c10961cf3a0b8d86ff024c5cb3740af694dd0e7f8d213b0c0 not found: ID does not exist" Oct 30 18:08:54.299336 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:54.299310 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm"] Oct 30 18:08:54.311635 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:54.311610 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-27567a-54b66d6688-gvqbm"] Oct 30 18:08:55.831251 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:55.831216 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:08:56.098392 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:56.098310 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9665073-6722-4274-bc2d-d453b06bd6c0" path="/var/lib/kubelet/pods/d9665073-6722-4274-bc2d-d453b06bd6c0/volumes" Oct 30 18:08:57.101414 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:57.101353 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-27567a-7c6f9f9747-hhm7h" Oct 30 18:08:57.259716 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:57.259684 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d-plugins-config-volume\") pod \"9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d\" (UID: \"9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d\") " Oct 30 18:08:57.259913 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:57.259794 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d-kube-api-access-gx2c5\" (UniqueName: \"kubernetes.io/projected/9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d-kube-api-access-gx2c5\") pod \"9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d\" (UID: \"9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d\") " Oct 30 18:08:57.260151 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:57.260127 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d-plugins-config-volume" pod "9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d" (UID: "9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:08:57.261979 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:57.261955 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d-kube-api-access-gx2c5" pod "9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d" (UID: "9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d"). InnerVolumeSpecName "kube-api-access-gx2c5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:08:57.267148 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:57.267120 4965 generic.go:356] "Generic (PLEG): container finished" podID="9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d" containerID="3ba69e2ce57bf5f1e104e01f625eb17e39014f25b3c2f7b5276f13d42a0a54a3" exitCode=0 Oct 30 18:08:57.267220 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:57.267166 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-27567a-7c6f9f9747-hhm7h" Oct 30 18:08:57.267220 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:57.267187 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-27567a-7c6f9f9747-hhm7h" event={"ID":"9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d","Type":"ContainerDied","Data":"3ba69e2ce57bf5f1e104e01f625eb17e39014f25b3c2f7b5276f13d42a0a54a3"} Oct 30 18:08:57.267302 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:57.267239 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-27567a-7c6f9f9747-hhm7h" event={"ID":"9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d","Type":"ContainerDied","Data":"4745739e47ab05dd18b31ab0e14cc22c45cbd8b85720a1d0e19cc5110961c9d4"} Oct 30 18:08:57.267302 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:57.267270 4965 scope.go:122] "RemoveContainer" containerID="3ba69e2ce57bf5f1e104e01f625eb17e39014f25b3c2f7b5276f13d42a0a54a3" Oct 30 18:08:57.276206 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:57.276190 4965 scope.go:122] "RemoveContainer" containerID="3ba69e2ce57bf5f1e104e01f625eb17e39014f25b3c2f7b5276f13d42a0a54a3" Oct 30 18:08:57.276496 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:08:57.276475 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ba69e2ce57bf5f1e104e01f625eb17e39014f25b3c2f7b5276f13d42a0a54a3\": container with ID starting with 3ba69e2ce57bf5f1e104e01f625eb17e39014f25b3c2f7b5276f13d42a0a54a3 not found: ID does not exist" containerID="3ba69e2ce57bf5f1e104e01f625eb17e39014f25b3c2f7b5276f13d42a0a54a3" Oct 30 18:08:57.276570 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:57.276506 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ba69e2ce57bf5f1e104e01f625eb17e39014f25b3c2f7b5276f13d42a0a54a3"} err="failed to get container status \"3ba69e2ce57bf5f1e104e01f625eb17e39014f25b3c2f7b5276f13d42a0a54a3\": rpc error: code = NotFound desc = could not find container \"3ba69e2ce57bf5f1e104e01f625eb17e39014f25b3c2f7b5276f13d42a0a54a3\": container with ID starting with 3ba69e2ce57bf5f1e104e01f625eb17e39014f25b3c2f7b5276f13d42a0a54a3 not found: ID does not exist" Oct 30 18:08:57.317493 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:57.317465 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-27567a-7c6f9f9747-hhm7h"] Oct 30 18:08:57.322972 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:57.322949 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-27567a-7c6f9f9747-hhm7h"] Oct 30 18:08:57.361263 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:57.361205 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:57.361263 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:57.361231 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx2c5\" (UniqueName: \"kubernetes.io/projected/9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d-kube-api-access-gx2c5\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:08:58.097999 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:08:58.097967 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d" path="/var/lib/kubelet/pods/9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d/volumes" Oct 30 18:09:01.385020 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:01.384988 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7"] Oct 30 18:09:05.053434 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:05.053392 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:09:05.995825 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:09:05.995777 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:09:16.406329 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:16.406283 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7" podUID="5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7" containerName="main" containerID="cri-o://d3eb557bc7658c75873df26175210dacc8c5c26a12ee53ecda5f6ef17478bc26" gracePeriod=45 Oct 30 18:09:16.661819 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:16.661765 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7" Oct 30 18:09:16.771655 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:16.771624 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-tls-certs\") pod \"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7\" (UID: \"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7\") " Oct 30 18:09:16.771833 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:16.771730 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-home\" (UniqueName: \"kubernetes.io/empty-dir/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-home\") pod \"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7\" (UID: \"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7\") " Oct 30 18:09:16.771833 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:16.771758 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-dshm\") pod \"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7\" (UID: \"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7\") " Oct 30 18:09:16.771833 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:16.771815 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-tmp-dir\") pod \"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7\" (UID: \"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7\") " Oct 30 18:09:16.772012 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:16.771851 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-model-cache\") pod \"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7\" (UID: \"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7\") " Oct 30 18:09:16.772012 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:16.771893 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-kube-api-access-sgh29\" (UniqueName: \"kubernetes.io/projected/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-kube-api-access-sgh29\") pod \"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7\" (UID: \"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7\") " Oct 30 18:09:16.772139 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:16.772029 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-home" pod "5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7" (UID: "5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:09:16.772213 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:16.772150 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-model-cache" pod "5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7" (UID: "5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:09:16.772213 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:16.772167 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-tmp-dir" pod "5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7" (UID: "5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:09:16.772374 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:16.772349 4965 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-model-cache\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:09:16.772374 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:16.772372 4965 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-home\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:09:16.772576 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:16.772383 4965 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-tmp-dir\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:09:16.773996 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:16.773976 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-dshm" pod "5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7" (UID: "5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:09:16.774095 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:16.774009 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-tls-certs" pod "5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7" (UID: "5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:09:16.774095 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:16.774008 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-kube-api-access-sgh29" pod "5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7" (UID: "5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7"). InnerVolumeSpecName "kube-api-access-sgh29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:09:16.873209 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:16.873176 4965 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-tls-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:09:16.873209 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:16.873202 4965 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-dshm\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:09:16.873209 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:16.873213 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgh29\" (UniqueName: \"kubernetes.io/projected/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7-kube-api-access-sgh29\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:09:17.375905 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:17.375857 4965 generic.go:356] "Generic (PLEG): container finished" podID="5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7" containerID="d3eb557bc7658c75873df26175210dacc8c5c26a12ee53ecda5f6ef17478bc26" exitCode=0 Oct 30 18:09:17.376085 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:17.375929 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7" Oct 30 18:09:17.376085 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:17.375959 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7" event={"ID":"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7","Type":"ContainerDied","Data":"d3eb557bc7658c75873df26175210dacc8c5c26a12ee53ecda5f6ef17478bc26"} Oct 30 18:09:17.376085 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:17.376006 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7" event={"ID":"5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7","Type":"ContainerDied","Data":"dbf52d828191147d272272ed911795c7cc0ee319b2a723adbe7a744a1955f642"} Oct 30 18:09:17.376085 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:17.376028 4965 scope.go:122] "RemoveContainer" containerID="d3eb557bc7658c75873df26175210dacc8c5c26a12ee53ecda5f6ef17478bc26" Oct 30 18:09:17.386064 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:17.386043 4965 scope.go:122] "RemoveContainer" containerID="d3eb557bc7658c75873df26175210dacc8c5c26a12ee53ecda5f6ef17478bc26" Oct 30 18:09:17.386348 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:09:17.386330 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3eb557bc7658c75873df26175210dacc8c5c26a12ee53ecda5f6ef17478bc26\": container with ID starting with d3eb557bc7658c75873df26175210dacc8c5c26a12ee53ecda5f6ef17478bc26 not found: ID does not exist" containerID="d3eb557bc7658c75873df26175210dacc8c5c26a12ee53ecda5f6ef17478bc26" Oct 30 18:09:17.386403 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:17.386357 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3eb557bc7658c75873df26175210dacc8c5c26a12ee53ecda5f6ef17478bc26"} err="failed to get container status \"d3eb557bc7658c75873df26175210dacc8c5c26a12ee53ecda5f6ef17478bc26\": rpc error: code = NotFound desc = could not find container \"d3eb557bc7658c75873df26175210dacc8c5c26a12ee53ecda5f6ef17478bc26\": container with ID starting with d3eb557bc7658c75873df26175210dacc8c5c26a12ee53ecda5f6ef17478bc26 not found: ID does not exist" Oct 30 18:09:17.428413 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:17.428380 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7"] Oct 30 18:09:17.440305 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:17.440280 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-95f55d/ipp-route-95f55d-kserve-64f7586f5c-fh9x7"] Oct 30 18:09:18.097448 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:18.097415 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7" path="/var/lib/kubelet/pods/5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7/volumes" Oct 30 18:09:35.996510 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:09:35.996472 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:09:45.193120 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:45.193087 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg"] Oct 30 18:09:45.527682 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:45.527582 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" podUID="4a2a43ad-860d-465a-a29a-130c83307e87" containerName="istio-proxy" containerID="cri-o://a086d9be149ff5d96c509f945f3ff144fe40962fe614676050145b9edecaff66" gracePeriod=30 Oct 30 18:09:50.687448 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:50.687401 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-222.ec2.internal" podUID="67161c7401d4577fb0911752be0668c2" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Oct 30 18:09:50.789324 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:50.789271 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:09:50.907587 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:50.907553 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-istio-data\") pod \"4a2a43ad-860d-465a-a29a-130c83307e87\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " Oct 30 18:09:50.907786 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:50.907625 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-istio-envoy\") pod \"4a2a43ad-860d-465a-a29a-130c83307e87\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " Oct 30 18:09:50.907786 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:50.907652 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a2a43ad-860d-465a-a29a-130c83307e87-kube-api-access-79pp9\" (UniqueName: \"kubernetes.io/projected/4a2a43ad-860d-465a-a29a-130c83307e87-kube-api-access-79pp9\") pod \"4a2a43ad-860d-465a-a29a-130c83307e87\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " Oct 30 18:09:50.907786 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:50.907694 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-workload-certs\") pod \"4a2a43ad-860d-465a-a29a-130c83307e87\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " Oct 30 18:09:50.907786 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:50.907714 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a2a43ad-860d-465a-a29a-130c83307e87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a2a43ad-860d-465a-a29a-130c83307e87-istio-podinfo\") pod \"4a2a43ad-860d-465a-a29a-130c83307e87\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " Oct 30 18:09:50.908072 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:50.907845 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-workload-socket\") pod \"4a2a43ad-860d-465a-a29a-130c83307e87\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " Oct 30 18:09:50.908072 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:50.907914 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a2a43ad-860d-465a-a29a-130c83307e87-istio-token\" (UniqueName: \"kubernetes.io/projected/4a2a43ad-860d-465a-a29a-130c83307e87-istio-token\") pod \"4a2a43ad-860d-465a-a29a-130c83307e87\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " Oct 30 18:09:50.908072 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:50.907930 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-istio-data" pod "4a2a43ad-860d-465a-a29a-130c83307e87" (UID: "4a2a43ad-860d-465a-a29a-130c83307e87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:09:50.908072 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:50.907963 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-credential-socket\") pod \"4a2a43ad-860d-465a-a29a-130c83307e87\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " Oct 30 18:09:50.908072 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:50.908020 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a2a43ad-860d-465a-a29a-130c83307e87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a2a43ad-860d-465a-a29a-130c83307e87-istiod-ca-cert\") pod \"4a2a43ad-860d-465a-a29a-130c83307e87\" (UID: \"4a2a43ad-860d-465a-a29a-130c83307e87\") " Oct 30 18:09:50.908360 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:50.908094 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-workload-certs" pod "4a2a43ad-860d-465a-a29a-130c83307e87" (UID: "4a2a43ad-860d-465a-a29a-130c83307e87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:09:50.908360 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:50.908139 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-workload-socket" pod "4a2a43ad-860d-465a-a29a-130c83307e87" (UID: "4a2a43ad-860d-465a-a29a-130c83307e87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:09:50.908360 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:50.908259 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-credential-socket" pod "4a2a43ad-860d-465a-a29a-130c83307e87" (UID: "4a2a43ad-860d-465a-a29a-130c83307e87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:09:50.908521 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:50.908455 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a2a43ad-860d-465a-a29a-130c83307e87-istiod-ca-cert" pod "4a2a43ad-860d-465a-a29a-130c83307e87" (UID: "4a2a43ad-860d-465a-a29a-130c83307e87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:09:50.908521 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:50.908479 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:09:50.908521 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:50.908495 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:09:50.908521 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:50.908505 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:09:50.908521 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:50.908514 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:09:50.910089 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:50.910064 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-istio-envoy" pod "4a2a43ad-860d-465a-a29a-130c83307e87" (UID: "4a2a43ad-860d-465a-a29a-130c83307e87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:09:50.910218 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:50.910199 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a2a43ad-860d-465a-a29a-130c83307e87-kube-api-access-79pp9" pod "4a2a43ad-860d-465a-a29a-130c83307e87" (UID: "4a2a43ad-860d-465a-a29a-130c83307e87"). InnerVolumeSpecName "kube-api-access-79pp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:09:50.910440 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:50.910416 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a2a43ad-860d-465a-a29a-130c83307e87-istio-token" pod "4a2a43ad-860d-465a-a29a-130c83307e87" (UID: "4a2a43ad-860d-465a-a29a-130c83307e87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:09:50.910541 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:50.910457 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a2a43ad-860d-465a-a29a-130c83307e87-istio-podinfo" pod "4a2a43ad-860d-465a-a29a-130c83307e87" (UID: "4a2a43ad-860d-465a-a29a-130c83307e87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:09:51.009500 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:51.009462 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a2a43ad-860d-465a-a29a-130c83307e87-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:09:51.009500 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:51.009491 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a2a43ad-860d-465a-a29a-130c83307e87-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:09:51.009500 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:51.009502 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a2a43ad-860d-465a-a29a-130c83307e87-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:09:51.009500 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:51.009511 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a2a43ad-860d-465a-a29a-130c83307e87-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:09:51.009771 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:51.009520 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79pp9\" (UniqueName: \"kubernetes.io/projected/4a2a43ad-860d-465a-a29a-130c83307e87-kube-api-access-79pp9\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:09:51.558840 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:51.558803 4965 generic.go:356] "Generic (PLEG): container finished" podID="4a2a43ad-860d-465a-a29a-130c83307e87" containerID="a086d9be149ff5d96c509f945f3ff144fe40962fe614676050145b9edecaff66" exitCode=0 Oct 30 18:09:51.559039 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:51.558854 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" Oct 30 18:09:51.559039 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:51.558892 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" event={"ID":"4a2a43ad-860d-465a-a29a-130c83307e87","Type":"ContainerDied","Data":"a086d9be149ff5d96c509f945f3ff144fe40962fe614676050145b9edecaff66"} Oct 30 18:09:51.559039 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:51.558948 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg" event={"ID":"4a2a43ad-860d-465a-a29a-130c83307e87","Type":"ContainerDied","Data":"c4a1d8dc4bc01e9036c033e51c76eae7ac621ec9d9e177043e1896e6da787d10"} Oct 30 18:09:51.559039 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:51.558969 4965 scope.go:122] "RemoveContainer" containerID="a086d9be149ff5d96c509f945f3ff144fe40962fe614676050145b9edecaff66" Oct 30 18:09:51.569146 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:51.569126 4965 scope.go:122] "RemoveContainer" containerID="bca2e1151789163153b4d37b2637190d8e61196448ac81819f62c347e45ca03e" Oct 30 18:09:51.577931 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:51.577914 4965 scope.go:122] "RemoveContainer" containerID="a086d9be149ff5d96c509f945f3ff144fe40962fe614676050145b9edecaff66" Oct 30 18:09:51.578183 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:09:51.578163 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a086d9be149ff5d96c509f945f3ff144fe40962fe614676050145b9edecaff66\": container with ID starting with a086d9be149ff5d96c509f945f3ff144fe40962fe614676050145b9edecaff66 not found: ID does not exist" containerID="a086d9be149ff5d96c509f945f3ff144fe40962fe614676050145b9edecaff66" Oct 30 18:09:51.578246 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:51.578192 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a086d9be149ff5d96c509f945f3ff144fe40962fe614676050145b9edecaff66"} err="failed to get container status \"a086d9be149ff5d96c509f945f3ff144fe40962fe614676050145b9edecaff66\": rpc error: code = NotFound desc = could not find container \"a086d9be149ff5d96c509f945f3ff144fe40962fe614676050145b9edecaff66\": container with ID starting with a086d9be149ff5d96c509f945f3ff144fe40962fe614676050145b9edecaff66 not found: ID does not exist" Oct 30 18:09:51.578246 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:51.578210 4965 scope.go:122] "RemoveContainer" containerID="bca2e1151789163153b4d37b2637190d8e61196448ac81819f62c347e45ca03e" Oct 30 18:09:51.578496 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:09:51.578475 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bca2e1151789163153b4d37b2637190d8e61196448ac81819f62c347e45ca03e\": container with ID starting with bca2e1151789163153b4d37b2637190d8e61196448ac81819f62c347e45ca03e not found: ID does not exist" containerID="bca2e1151789163153b4d37b2637190d8e61196448ac81819f62c347e45ca03e" Oct 30 18:09:51.578549 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:51.578502 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca2e1151789163153b4d37b2637190d8e61196448ac81819f62c347e45ca03e"} err="failed to get container status \"bca2e1151789163153b4d37b2637190d8e61196448ac81819f62c347e45ca03e\": rpc error: code = NotFound desc = could not find container \"bca2e1151789163153b4d37b2637190d8e61196448ac81819f62c347e45ca03e\": container with ID starting with bca2e1151789163153b4d37b2637190d8e61196448ac81819f62c347e45ca03e not found: ID does not exist" Oct 30 18:09:51.622367 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:51.622337 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg"] Oct 30 18:09:51.634162 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:51.634135 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-95f55d-openshift-default-9fb57df5-xjdhg"] Oct 30 18:09:52.097107 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:52.097070 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a2a43ad-860d-465a-a29a-130c83307e87" path="/var/lib/kubelet/pods/4a2a43ad-860d-465a-a29a-130c83307e87/volumes" Oct 30 18:09:59.423033 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:09:59.422999 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:10:05.996550 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:10:05.996468 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:10:28.056439 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:28.056404 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g"] Oct 30 18:10:28.057003 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:28.056665 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g" podUID="40e1abc1-ddb7-44f3-b112-e704e1ebbc81" containerName="maas-api" containerID="cri-o://e413056d747329120e70b395e491aefa0f47dd5f131ce930c01e76078739f8b4" gracePeriod=30 Oct 30 18:10:28.335434 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:28.335404 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g" Oct 30 18:10:28.388397 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:28.388366 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40e1abc1-ddb7-44f3-b112-e704e1ebbc81-metrics-certs\" (UniqueName: \"kubernetes.io/secret/40e1abc1-ddb7-44f3-b112-e704e1ebbc81-metrics-certs\") pod \"40e1abc1-ddb7-44f3-b112-e704e1ebbc81\" (UID: \"40e1abc1-ddb7-44f3-b112-e704e1ebbc81\") " Oct 30 18:10:28.388585 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:28.388445 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40e1abc1-ddb7-44f3-b112-e704e1ebbc81-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40e1abc1-ddb7-44f3-b112-e704e1ebbc81-openshift-service-ca\") pod \"40e1abc1-ddb7-44f3-b112-e704e1ebbc81\" (UID: \"40e1abc1-ddb7-44f3-b112-e704e1ebbc81\") " Oct 30 18:10:28.388585 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:28.388543 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40e1abc1-ddb7-44f3-b112-e704e1ebbc81-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40e1abc1-ddb7-44f3-b112-e704e1ebbc81-maas-api-tls\") pod \"40e1abc1-ddb7-44f3-b112-e704e1ebbc81\" (UID: \"40e1abc1-ddb7-44f3-b112-e704e1ebbc81\") " Oct 30 18:10:28.388585 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:28.388573 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40e1abc1-ddb7-44f3-b112-e704e1ebbc81-kube-api-access-k8fpg\" (UniqueName: \"kubernetes.io/projected/40e1abc1-ddb7-44f3-b112-e704e1ebbc81-kube-api-access-k8fpg\") pod \"40e1abc1-ddb7-44f3-b112-e704e1ebbc81\" (UID: \"40e1abc1-ddb7-44f3-b112-e704e1ebbc81\") " Oct 30 18:10:28.388904 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:28.388878 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e1abc1-ddb7-44f3-b112-e704e1ebbc81-openshift-service-ca" pod "40e1abc1-ddb7-44f3-b112-e704e1ebbc81" (UID: "40e1abc1-ddb7-44f3-b112-e704e1ebbc81"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:10:28.390728 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:28.390703 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40e1abc1-ddb7-44f3-b112-e704e1ebbc81-maas-api-tls" pod "40e1abc1-ddb7-44f3-b112-e704e1ebbc81" (UID: "40e1abc1-ddb7-44f3-b112-e704e1ebbc81"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:10:28.391291 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:28.391263 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e1abc1-ddb7-44f3-b112-e704e1ebbc81-kube-api-access-k8fpg" pod "40e1abc1-ddb7-44f3-b112-e704e1ebbc81" (UID: "40e1abc1-ddb7-44f3-b112-e704e1ebbc81"). InnerVolumeSpecName "kube-api-access-k8fpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:10:28.392175 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:28.392150 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40e1abc1-ddb7-44f3-b112-e704e1ebbc81-metrics-certs" pod "40e1abc1-ddb7-44f3-b112-e704e1ebbc81" (UID: "40e1abc1-ddb7-44f3-b112-e704e1ebbc81"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:10:28.491739 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:28.491695 4965 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40e1abc1-ddb7-44f3-b112-e704e1ebbc81-metrics-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:10:28.491739 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:28.491739 4965 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40e1abc1-ddb7-44f3-b112-e704e1ebbc81-openshift-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:10:28.492217 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:28.491757 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40e1abc1-ddb7-44f3-b112-e704e1ebbc81-maas-api-tls\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:10:28.492217 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:28.491770 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8fpg\" (UniqueName: \"kubernetes.io/projected/40e1abc1-ddb7-44f3-b112-e704e1ebbc81-kube-api-access-k8fpg\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:10:28.744554 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:28.744521 4965 generic.go:356] "Generic (PLEG): container finished" podID="40e1abc1-ddb7-44f3-b112-e704e1ebbc81" containerID="e413056d747329120e70b395e491aefa0f47dd5f131ce930c01e76078739f8b4" exitCode=0 Oct 30 18:10:28.744773 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:28.744571 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g" Oct 30 18:10:28.744773 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:28.744619 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g" event={"ID":"40e1abc1-ddb7-44f3-b112-e704e1ebbc81","Type":"ContainerDied","Data":"e413056d747329120e70b395e491aefa0f47dd5f131ce930c01e76078739f8b4"} Oct 30 18:10:28.744773 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:28.744681 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g" event={"ID":"40e1abc1-ddb7-44f3-b112-e704e1ebbc81","Type":"ContainerDied","Data":"6cec52d3a932f86569779fcc2b64486b6944432532b7de15651ad47e803779c4"} Oct 30 18:10:28.744773 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:28.744713 4965 scope.go:122] "RemoveContainer" containerID="e413056d747329120e70b395e491aefa0f47dd5f131ce930c01e76078739f8b4" Oct 30 18:10:28.757550 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:28.757532 4965 scope.go:122] "RemoveContainer" containerID="e413056d747329120e70b395e491aefa0f47dd5f131ce930c01e76078739f8b4" Oct 30 18:10:28.758342 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:10:28.758313 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e413056d747329120e70b395e491aefa0f47dd5f131ce930c01e76078739f8b4\": container with ID starting with e413056d747329120e70b395e491aefa0f47dd5f131ce930c01e76078739f8b4 not found: ID does not exist" containerID="e413056d747329120e70b395e491aefa0f47dd5f131ce930c01e76078739f8b4" Oct 30 18:10:28.758418 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:28.758355 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e413056d747329120e70b395e491aefa0f47dd5f131ce930c01e76078739f8b4"} err="failed to get container status \"e413056d747329120e70b395e491aefa0f47dd5f131ce930c01e76078739f8b4\": rpc error: code = NotFound desc = could not find container \"e413056d747329120e70b395e491aefa0f47dd5f131ce930c01e76078739f8b4\": container with ID starting with e413056d747329120e70b395e491aefa0f47dd5f131ce930c01e76078739f8b4 not found: ID does not exist" Oct 30 18:10:28.794975 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:28.794947 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g"] Oct 30 18:10:28.799098 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:28.799078 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a2a6cb-9474b7dd-5hs8g"] Oct 30 18:10:30.098546 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:30.098514 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40e1abc1-ddb7-44f3-b112-e704e1ebbc81" path="/var/lib/kubelet/pods/40e1abc1-ddb7-44f3-b112-e704e1ebbc81/volumes" Oct 30 18:10:31.346937 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:31.346900 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:10:35.652205 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:35.652165 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn"] Oct 30 18:10:35.652718 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:35.652453 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" podUID="ef413002-b509-4658-a123-a5b4b0715d58" containerName="istio-proxy" containerID="cri-o://7e36fc6874820207e79e41512526c66c0a1ce2c376bb3908150814c6a42a9084" gracePeriod=30 Oct 30 18:10:35.995846 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:10:35.995761 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:10:36.173678 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.173646 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a23c98-j9tcq"] Oct 30 18:10:36.174077 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.174064 4965 cpu_manager.go:414] "containerMap: removing container" podUID="9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d" containerName="payload-pre-processing" Oct 30 18:10:36.174131 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.174078 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d" containerName="payload-pre-processing" Oct 30 18:10:36.174131 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.174106 4965 cpu_manager.go:414] "containerMap: removing container" podUID="6497147f-bad2-434c-a6e4-45b247b75fc2" containerName="istio-proxy" Oct 30 18:10:36.174131 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.174111 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6497147f-bad2-434c-a6e4-45b247b75fc2" containerName="istio-proxy" Oct 30 18:10:36.174131 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.174118 4965 cpu_manager.go:414] "containerMap: removing container" podUID="5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7" containerName="main" Oct 30 18:10:36.174131 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.174124 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7" containerName="main" Oct 30 18:10:36.174302 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.174135 4965 cpu_manager.go:414] "containerMap: removing container" podUID="40e1abc1-ddb7-44f3-b112-e704e1ebbc81" containerName="maas-api" Oct 30 18:10:36.174302 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.174140 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40e1abc1-ddb7-44f3-b112-e704e1ebbc81" containerName="maas-api" Oct 30 18:10:36.174302 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.174152 4965 cpu_manager.go:414] "containerMap: removing container" podUID="4a2a43ad-860d-465a-a29a-130c83307e87" containerName="istio-proxy" Oct 30 18:10:36.174302 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.174157 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a2a43ad-860d-465a-a29a-130c83307e87" containerName="istio-proxy" Oct 30 18:10:36.174302 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.174165 4965 cpu_manager.go:414] "containerMap: removing container" podUID="d9665073-6722-4274-bc2d-d453b06bd6c0" containerName="maas-api" Oct 30 18:10:36.174302 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.174170 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9665073-6722-4274-bc2d-d453b06bd6c0" containerName="maas-api" Oct 30 18:10:36.174302 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.174181 4965 cpu_manager.go:414] "containerMap: removing container" podUID="4a2a43ad-860d-465a-a29a-130c83307e87" containerName="istio-proxy" Oct 30 18:10:36.174302 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.174186 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a2a43ad-860d-465a-a29a-130c83307e87" containerName="istio-proxy" Oct 30 18:10:36.174302 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.174192 4965 cpu_manager.go:414] "containerMap: removing container" podUID="93b77541-e76a-4b77-99f7-68e1e5e2a624" containerName="revoke-keys" Oct 30 18:10:36.174302 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.174197 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93b77541-e76a-4b77-99f7-68e1e5e2a624" containerName="revoke-keys" Oct 30 18:10:36.174302 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.174268 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="93b77541-e76a-4b77-99f7-68e1e5e2a624" containerName="revoke-keys" Oct 30 18:10:36.174302 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.174280 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a4c9a42-ca45-4f5b-96c3-6b3bad58f62d" containerName="payload-pre-processing" Oct 30 18:10:36.174302 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.174289 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e8b2094-be3e-4a33-92a5-7e2ca6e0b8c7" containerName="main" Oct 30 18:10:36.174302 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.174295 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="40e1abc1-ddb7-44f3-b112-e704e1ebbc81" containerName="maas-api" Oct 30 18:10:36.174302 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.174302 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a2a43ad-860d-465a-a29a-130c83307e87" containerName="istio-proxy" Oct 30 18:10:36.174302 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.174308 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9665073-6722-4274-bc2d-d453b06bd6c0" containerName="maas-api" Oct 30 18:10:36.174792 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.174313 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="6497147f-bad2-434c-a6e4-45b247b75fc2" containerName="istio-proxy" Oct 30 18:10:36.174792 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.174319 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a2a43ad-860d-465a-a29a-130c83307e87" containerName="istio-proxy" Oct 30 18:10:36.178817 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.178801 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a23c98-j9tcq" Oct 30 18:10:36.196002 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.195978 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a23c98-j9tcq"] Oct 30 18:10:36.221780 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.221749 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\"" Oct 30 18:10:36.221780 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.221757 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:10:36.271361 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.271296 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p86r\" (UniqueName: \"kubernetes.io/projected/065dad47-8b1a-4b1c-87df-80e3f106e54c-kube-api-access-9p86r\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a23c98-j9tcq\" (UID: \"065dad47-8b1a-4b1c-87df-80e3f106e54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a23c98-j9tcq" Oct 30 18:10:36.271361 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.271350 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/065dad47-8b1a-4b1c-87df-80e3f106e54c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a23c98-j9tcq\" (UID: \"065dad47-8b1a-4b1c-87df-80e3f106e54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a23c98-j9tcq" Oct 30 18:10:36.271583 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.271453 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/065dad47-8b1a-4b1c-87df-80e3f106e54c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a23c98-j9tcq\" (UID: \"065dad47-8b1a-4b1c-87df-80e3f106e54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a23c98-j9tcq" Oct 30 18:10:36.372994 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.372959 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p86r\" (UniqueName: \"kubernetes.io/projected/065dad47-8b1a-4b1c-87df-80e3f106e54c-kube-api-access-9p86r\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a23c98-j9tcq\" (UID: \"065dad47-8b1a-4b1c-87df-80e3f106e54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a23c98-j9tcq" Oct 30 18:10:36.373199 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.373018 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/065dad47-8b1a-4b1c-87df-80e3f106e54c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a23c98-j9tcq\" (UID: \"065dad47-8b1a-4b1c-87df-80e3f106e54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a23c98-j9tcq" Oct 30 18:10:36.373199 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.373054 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/065dad47-8b1a-4b1c-87df-80e3f106e54c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a23c98-j9tcq\" (UID: \"065dad47-8b1a-4b1c-87df-80e3f106e54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a23c98-j9tcq" Oct 30 18:10:36.373824 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.373803 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/065dad47-8b1a-4b1c-87df-80e3f106e54c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a23c98-j9tcq\" (UID: \"065dad47-8b1a-4b1c-87df-80e3f106e54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a23c98-j9tcq" Oct 30 18:10:36.375475 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.375457 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/065dad47-8b1a-4b1c-87df-80e3f106e54c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a23c98-j9tcq\" (UID: \"065dad47-8b1a-4b1c-87df-80e3f106e54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a23c98-j9tcq" Oct 30 18:10:36.395885 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.395833 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p86r\" (UniqueName: \"kubernetes.io/projected/065dad47-8b1a-4b1c-87df-80e3f106e54c-kube-api-access-9p86r\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a23c98-j9tcq\" (UID: \"065dad47-8b1a-4b1c-87df-80e3f106e54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a23c98-j9tcq" Oct 30 18:10:36.491938 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.491901 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a23c98-j9tcq" Oct 30 18:10:36.790226 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:36.790201 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a23c98-j9tcq"] Oct 30 18:10:36.792216 ip-10-0-139-222 kubenswrapper[4965]: W1030 18:10:36.792141 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod065dad47_8b1a_4b1c_87df_80e3f106e54c.slice/crio-9d5f6933ad42f7ae3a3aaa59400813c0d7a61d015dc1c110742126674ebfc5b0 WatchSource:0}: Error finding container 9d5f6933ad42f7ae3a3aaa59400813c0d7a61d015dc1c110742126674ebfc5b0: Status 404 returned error can't find the container with id 9d5f6933ad42f7ae3a3aaa59400813c0d7a61d015dc1c110742126674ebfc5b0 Oct 30 18:10:37.790465 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:37.790433 4965 generic.go:356] "Generic (PLEG): container finished" podID="065dad47-8b1a-4b1c-87df-80e3f106e54c" containerID="7dcc2c0941e016a529075d13dc131314a4014699394b228a6df7cd0ae08de9b6" exitCode=0 Oct 30 18:10:37.790923 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:37.790517 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a23c98-j9tcq" event={"ID":"065dad47-8b1a-4b1c-87df-80e3f106e54c","Type":"ContainerDied","Data":"7dcc2c0941e016a529075d13dc131314a4014699394b228a6df7cd0ae08de9b6"} Oct 30 18:10:37.790923 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:37.790567 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a23c98-j9tcq" event={"ID":"065dad47-8b1a-4b1c-87df-80e3f106e54c","Type":"ContainerStarted","Data":"9d5f6933ad42f7ae3a3aaa59400813c0d7a61d015dc1c110742126674ebfc5b0"} Oct 30 18:10:38.925053 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:38.925027 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a23c98-j9tcq" Oct 30 18:10:38.997101 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:38.997068 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/065dad47-8b1a-4b1c-87df-80e3f106e54c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/065dad47-8b1a-4b1c-87df-80e3f106e54c-maas-api-openshift-ca\") pod \"065dad47-8b1a-4b1c-87df-80e3f106e54c\" (UID: \"065dad47-8b1a-4b1c-87df-80e3f106e54c\") " Oct 30 18:10:38.997295 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:38.997133 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/065dad47-8b1a-4b1c-87df-80e3f106e54c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/065dad47-8b1a-4b1c-87df-80e3f106e54c-maas-api-xks-ca\") pod \"065dad47-8b1a-4b1c-87df-80e3f106e54c\" (UID: \"065dad47-8b1a-4b1c-87df-80e3f106e54c\") " Oct 30 18:10:38.997295 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:38.997215 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/065dad47-8b1a-4b1c-87df-80e3f106e54c-kube-api-access-9p86r\" (UniqueName: \"kubernetes.io/projected/065dad47-8b1a-4b1c-87df-80e3f106e54c-kube-api-access-9p86r\") pod \"065dad47-8b1a-4b1c-87df-80e3f106e54c\" (UID: \"065dad47-8b1a-4b1c-87df-80e3f106e54c\") " Oct 30 18:10:38.997506 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:38.997479 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/065dad47-8b1a-4b1c-87df-80e3f106e54c-maas-api-openshift-ca" pod "065dad47-8b1a-4b1c-87df-80e3f106e54c" (UID: "065dad47-8b1a-4b1c-87df-80e3f106e54c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:10:38.999315 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:38.999287 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/065dad47-8b1a-4b1c-87df-80e3f106e54c-maas-api-xks-ca" pod "065dad47-8b1a-4b1c-87df-80e3f106e54c" (UID: "065dad47-8b1a-4b1c-87df-80e3f106e54c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:10:38.999428 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:38.999374 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/065dad47-8b1a-4b1c-87df-80e3f106e54c-kube-api-access-9p86r" pod "065dad47-8b1a-4b1c-87df-80e3f106e54c" (UID: "065dad47-8b1a-4b1c-87df-80e3f106e54c"). InnerVolumeSpecName "kube-api-access-9p86r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:10:39.098362 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:39.098331 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p86r\" (UniqueName: \"kubernetes.io/projected/065dad47-8b1a-4b1c-87df-80e3f106e54c-kube-api-access-9p86r\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:10:39.098362 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:39.098362 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/065dad47-8b1a-4b1c-87df-80e3f106e54c-maas-api-openshift-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:10:39.098554 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:39.098375 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/065dad47-8b1a-4b1c-87df-80e3f106e54c-maas-api-xks-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:10:39.802209 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:39.802178 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a23c98-j9tcq" Oct 30 18:10:39.802373 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:39.802217 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a23c98-j9tcq" event={"ID":"065dad47-8b1a-4b1c-87df-80e3f106e54c","Type":"ContainerDied","Data":"9d5f6933ad42f7ae3a3aaa59400813c0d7a61d015dc1c110742126674ebfc5b0"} Oct 30 18:10:39.802373 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:39.802273 4965 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d5f6933ad42f7ae3a3aaa59400813c0d7a61d015dc1c110742126674ebfc5b0" Oct 30 18:10:40.810675 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:40.810643 4965 generic.go:356] "Generic (PLEG): container finished" podID="ef413002-b509-4658-a123-a5b4b0715d58" containerID="7e36fc6874820207e79e41512526c66c0a1ce2c376bb3908150814c6a42a9084" exitCode=0 Oct 30 18:10:40.811084 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:40.810714 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" event={"ID":"ef413002-b509-4658-a123-a5b4b0715d58","Type":"ContainerDied","Data":"7e36fc6874820207e79e41512526c66c0a1ce2c376bb3908150814c6a42a9084"} Oct 30 18:10:40.919778 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:40.919716 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 18:10:41.023781 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:41.023748 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef413002-b509-4658-a123-a5b4b0715d58-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef413002-b509-4658-a123-a5b4b0715d58-istiod-ca-cert\") pod \"ef413002-b509-4658-a123-a5b4b0715d58\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " Oct 30 18:10:41.023955 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:41.023830 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-istio-envoy\") pod \"ef413002-b509-4658-a123-a5b4b0715d58\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " Oct 30 18:10:41.023955 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:41.023930 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef413002-b509-4658-a123-a5b4b0715d58-kube-api-access-fdjsn\" (UniqueName: \"kubernetes.io/projected/ef413002-b509-4658-a123-a5b4b0715d58-kube-api-access-fdjsn\") pod \"ef413002-b509-4658-a123-a5b4b0715d58\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " Oct 30 18:10:41.024052 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:41.023989 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef413002-b509-4658-a123-a5b4b0715d58-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef413002-b509-4658-a123-a5b4b0715d58-istio-podinfo\") pod \"ef413002-b509-4658-a123-a5b4b0715d58\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " Oct 30 18:10:41.024052 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:41.024010 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-workload-socket\") pod \"ef413002-b509-4658-a123-a5b4b0715d58\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " Oct 30 18:10:41.024052 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:41.024037 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-istio-data\") pod \"ef413002-b509-4658-a123-a5b4b0715d58\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " Oct 30 18:10:41.024220 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:41.024057 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-workload-certs\") pod \"ef413002-b509-4658-a123-a5b4b0715d58\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " Oct 30 18:10:41.024220 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:41.024103 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-credential-socket\") pod \"ef413002-b509-4658-a123-a5b4b0715d58\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " Oct 30 18:10:41.024220 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:41.024126 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef413002-b509-4658-a123-a5b4b0715d58-istiod-ca-cert" pod "ef413002-b509-4658-a123-a5b4b0715d58" (UID: "ef413002-b509-4658-a123-a5b4b0715d58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:10:41.024220 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:41.024132 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef413002-b509-4658-a123-a5b4b0715d58-istio-token\" (UniqueName: \"kubernetes.io/projected/ef413002-b509-4658-a123-a5b4b0715d58-istio-token\") pod \"ef413002-b509-4658-a123-a5b4b0715d58\" (UID: \"ef413002-b509-4658-a123-a5b4b0715d58\") " Oct 30 18:10:41.024448 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:41.024326 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-workload-socket" pod "ef413002-b509-4658-a123-a5b4b0715d58" (UID: "ef413002-b509-4658-a123-a5b4b0715d58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:10:41.024513 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:41.024457 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-workload-certs" pod "ef413002-b509-4658-a123-a5b4b0715d58" (UID: "ef413002-b509-4658-a123-a5b4b0715d58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:10:41.024595 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:41.024545 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-credential-socket" pod "ef413002-b509-4658-a123-a5b4b0715d58" (UID: "ef413002-b509-4658-a123-a5b4b0715d58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:10:41.024595 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:41.024584 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-istio-data" pod "ef413002-b509-4658-a123-a5b4b0715d58" (UID: "ef413002-b509-4658-a123-a5b4b0715d58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:10:41.024778 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:41.024757 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:10:41.024829 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:41.024783 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:10:41.024829 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:41.024797 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:10:41.024829 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:41.024811 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef413002-b509-4658-a123-a5b4b0715d58-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:10:41.026268 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:41.026242 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef413002-b509-4658-a123-a5b4b0715d58-istio-podinfo" pod "ef413002-b509-4658-a123-a5b4b0715d58" (UID: "ef413002-b509-4658-a123-a5b4b0715d58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:10:41.026268 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:41.026250 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef413002-b509-4658-a123-a5b4b0715d58-istio-token" pod "ef413002-b509-4658-a123-a5b4b0715d58" (UID: "ef413002-b509-4658-a123-a5b4b0715d58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:10:41.026555 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:41.026536 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef413002-b509-4658-a123-a5b4b0715d58-kube-api-access-fdjsn" pod "ef413002-b509-4658-a123-a5b4b0715d58" (UID: "ef413002-b509-4658-a123-a5b4b0715d58"). InnerVolumeSpecName "kube-api-access-fdjsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:10:41.026555 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:41.026538 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-istio-envoy" pod "ef413002-b509-4658-a123-a5b4b0715d58" (UID: "ef413002-b509-4658-a123-a5b4b0715d58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:10:41.125497 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:41.125458 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:10:41.125497 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:41.125489 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef413002-b509-4658-a123-a5b4b0715d58-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:10:41.125497 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:41.125501 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef413002-b509-4658-a123-a5b4b0715d58-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:10:41.125717 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:41.125513 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdjsn\" (UniqueName: \"kubernetes.io/projected/ef413002-b509-4658-a123-a5b4b0715d58-kube-api-access-fdjsn\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:10:41.125717 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:41.125529 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef413002-b509-4658-a123-a5b4b0715d58-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:10:41.818003 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:41.817965 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" event={"ID":"ef413002-b509-4658-a123-a5b4b0715d58","Type":"ContainerDied","Data":"f1caf9382008cb6b8afaf0920e6a953e3848e38cd439a52689b0dbfa05772fed"} Oct 30 18:10:41.818485 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:41.818007 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn" Oct 30 18:10:41.818485 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:41.818040 4965 scope.go:122] "RemoveContainer" containerID="7e36fc6874820207e79e41512526c66c0a1ce2c376bb3908150814c6a42a9084" Oct 30 18:10:41.881300 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:41.881271 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn"] Oct 30 18:10:41.893353 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:41.893329 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a2a6cb-openshift-default-79585f7ddf-6jbtn"] Oct 30 18:10:42.097367 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:42.097334 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef413002-b509-4658-a123-a5b4b0715d58" path="/var/lib/kubelet/pods/ef413002-b509-4658-a123-a5b4b0715d58/volumes" Oct 30 18:10:46.618972 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:46.618938 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a23c98-787c476954-j5ls2"] Oct 30 18:10:46.645603 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:46.645571 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a23c98-76b6d5c87-8tx7q"] Oct 30 18:10:48.384412 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:48.384378 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a23c98-j9tcq"] Oct 30 18:10:48.396590 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:48.396561 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a23c98-j9tcq"] Oct 30 18:10:50.097131 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:50.097100 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="065dad47-8b1a-4b1c-87df-80e3f106e54c" path="/var/lib/kubelet/pods/065dad47-8b1a-4b1c-87df-80e3f106e54c/volumes" Oct 30 18:10:50.843956 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:50.843922 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:10:51.620007 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:51.619968 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a23c98-787c476954-j5ls2" podUID="b6b5b4db-cdc6-4d3d-863e-678a337f87bb" containerName="payload-processing" containerID="cri-o://9461c21293a94982a23cbb1c06ecaf6cba2ed5b2a778ab5b3c646594d43eeb5e" gracePeriod=25 Oct 30 18:10:51.646786 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:10:51.646753 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a23c98-76b6d5c87-8tx7q" podUID="dea747ba-4708-4307-8744-911dea32712e" containerName="payload-pre-processing" containerID="cri-o://3269f404e9274f8c954539743ffe790cdeb631032e21c1a94f3cb5ca6e0d02c9" gracePeriod=25 Oct 30 18:11:00.261853 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:00.261808 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a23c98-76b6d5c87-8tx7q" podUID="dea747ba-4708-4307-8744-911dea32712e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:11:00.261853 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:00.261808 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a23c98-787c476954-j5ls2" podUID="b6b5b4db-cdc6-4d3d-863e-678a337f87bb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:11:01.095319 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:01.095284 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n"] Oct 30 18:11:05.995890 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:11:05.995835 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:11:10.261551 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:10.261502 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a23c98-76b6d5c87-8tx7q" podUID="dea747ba-4708-4307-8744-911dea32712e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:11:10.261981 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:10.261519 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a23c98-787c476954-j5ls2" podUID="b6b5b4db-cdc6-4d3d-863e-678a337f87bb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:11:11.913270 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:11.913242 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a23c98-787c476954-j5ls2" Oct 30 18:11:11.933139 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:11.933113 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a23c98-76b6d5c87-8tx7q" Oct 30 18:11:11.984245 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:11.984211 4965 generic.go:356] "Generic (PLEG): container finished" podID="b6b5b4db-cdc6-4d3d-863e-678a337f87bb" containerID="9461c21293a94982a23cbb1c06ecaf6cba2ed5b2a778ab5b3c646594d43eeb5e" exitCode=0 Oct 30 18:11:11.984433 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:11.984258 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a23c98-787c476954-j5ls2" Oct 30 18:11:11.984433 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:11.984295 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a23c98-787c476954-j5ls2" event={"ID":"b6b5b4db-cdc6-4d3d-863e-678a337f87bb","Type":"ContainerDied","Data":"9461c21293a94982a23cbb1c06ecaf6cba2ed5b2a778ab5b3c646594d43eeb5e"} Oct 30 18:11:11.984433 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:11.984349 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a23c98-787c476954-j5ls2" event={"ID":"b6b5b4db-cdc6-4d3d-863e-678a337f87bb","Type":"ContainerDied","Data":"1fd12dc8029944ae02765e9c75e5993e1f458c715abf97a0076d677328dc1eb2"} Oct 30 18:11:11.984433 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:11.984377 4965 scope.go:122] "RemoveContainer" containerID="9461c21293a94982a23cbb1c06ecaf6cba2ed5b2a778ab5b3c646594d43eeb5e" Oct 30 18:11:11.985664 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:11.985640 4965 generic.go:356] "Generic (PLEG): container finished" podID="dea747ba-4708-4307-8744-911dea32712e" containerID="3269f404e9274f8c954539743ffe790cdeb631032e21c1a94f3cb5ca6e0d02c9" exitCode=0 Oct 30 18:11:11.985778 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:11.985674 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a23c98-76b6d5c87-8tx7q" Oct 30 18:11:11.985778 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:11.985672 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a23c98-76b6d5c87-8tx7q" event={"ID":"dea747ba-4708-4307-8744-911dea32712e","Type":"ContainerDied","Data":"3269f404e9274f8c954539743ffe790cdeb631032e21c1a94f3cb5ca6e0d02c9"} Oct 30 18:11:11.985886 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:11.985786 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a23c98-76b6d5c87-8tx7q" event={"ID":"dea747ba-4708-4307-8744-911dea32712e","Type":"ContainerDied","Data":"f680b65939013c1c412d400f8f19d443ecf078b5a38fc8a12af0598f4e328bf2"} Oct 30 18:11:11.995083 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:11.995066 4965 scope.go:122] "RemoveContainer" containerID="9461c21293a94982a23cbb1c06ecaf6cba2ed5b2a778ab5b3c646594d43eeb5e" Oct 30 18:11:11.995364 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:11:11.995344 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9461c21293a94982a23cbb1c06ecaf6cba2ed5b2a778ab5b3c646594d43eeb5e\": container with ID starting with 9461c21293a94982a23cbb1c06ecaf6cba2ed5b2a778ab5b3c646594d43eeb5e not found: ID does not exist" containerID="9461c21293a94982a23cbb1c06ecaf6cba2ed5b2a778ab5b3c646594d43eeb5e" Oct 30 18:11:11.995455 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:11.995376 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9461c21293a94982a23cbb1c06ecaf6cba2ed5b2a778ab5b3c646594d43eeb5e"} err="failed to get container status \"9461c21293a94982a23cbb1c06ecaf6cba2ed5b2a778ab5b3c646594d43eeb5e\": rpc error: code = NotFound desc = could not find container \"9461c21293a94982a23cbb1c06ecaf6cba2ed5b2a778ab5b3c646594d43eeb5e\": container with ID starting with 9461c21293a94982a23cbb1c06ecaf6cba2ed5b2a778ab5b3c646594d43eeb5e not found: ID does not exist" Oct 30 18:11:11.995455 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:11.995398 4965 scope.go:122] "RemoveContainer" containerID="3269f404e9274f8c954539743ffe790cdeb631032e21c1a94f3cb5ca6e0d02c9" Oct 30 18:11:12.004829 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:12.004803 4965 scope.go:122] "RemoveContainer" containerID="3269f404e9274f8c954539743ffe790cdeb631032e21c1a94f3cb5ca6e0d02c9" Oct 30 18:11:12.005117 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:11:12.005093 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3269f404e9274f8c954539743ffe790cdeb631032e21c1a94f3cb5ca6e0d02c9\": container with ID starting with 3269f404e9274f8c954539743ffe790cdeb631032e21c1a94f3cb5ca6e0d02c9 not found: ID does not exist" containerID="3269f404e9274f8c954539743ffe790cdeb631032e21c1a94f3cb5ca6e0d02c9" Oct 30 18:11:12.005209 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:12.005122 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3269f404e9274f8c954539743ffe790cdeb631032e21c1a94f3cb5ca6e0d02c9"} err="failed to get container status \"3269f404e9274f8c954539743ffe790cdeb631032e21c1a94f3cb5ca6e0d02c9\": rpc error: code = NotFound desc = could not find container \"3269f404e9274f8c954539743ffe790cdeb631032e21c1a94f3cb5ca6e0d02c9\": container with ID starting with 3269f404e9274f8c954539743ffe790cdeb631032e21c1a94f3cb5ca6e0d02c9 not found: ID does not exist" Oct 30 18:11:12.023518 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:12.023499 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dea747ba-4708-4307-8744-911dea32712e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dea747ba-4708-4307-8744-911dea32712e-plugins-config-volume\") pod \"dea747ba-4708-4307-8744-911dea32712e\" (UID: \"dea747ba-4708-4307-8744-911dea32712e\") " Oct 30 18:11:12.023653 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:12.023553 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dea747ba-4708-4307-8744-911dea32712e-kube-api-access-rj957\" (UniqueName: \"kubernetes.io/projected/dea747ba-4708-4307-8744-911dea32712e-kube-api-access-rj957\") pod \"dea747ba-4708-4307-8744-911dea32712e\" (UID: \"dea747ba-4708-4307-8744-911dea32712e\") " Oct 30 18:11:12.023653 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:12.023582 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6b5b4db-cdc6-4d3d-863e-678a337f87bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b5b4db-cdc6-4d3d-863e-678a337f87bb-plugins-config-volume\") pod \"b6b5b4db-cdc6-4d3d-863e-678a337f87bb\" (UID: \"b6b5b4db-cdc6-4d3d-863e-678a337f87bb\") " Oct 30 18:11:12.023653 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:12.023652 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b5b4db-cdc6-4d3d-863e-678a337f87bb-kube-api-access-qrlqw\" (UniqueName: \"kubernetes.io/projected/b6b5b4db-cdc6-4d3d-863e-678a337f87bb-kube-api-access-qrlqw\") pod \"b6b5b4db-cdc6-4d3d-863e-678a337f87bb\" (UID: \"b6b5b4db-cdc6-4d3d-863e-678a337f87bb\") " Oct 30 18:11:12.024056 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:12.024032 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b5b4db-cdc6-4d3d-863e-678a337f87bb-plugins-config-volume" pod "b6b5b4db-cdc6-4d3d-863e-678a337f87bb" (UID: "b6b5b4db-cdc6-4d3d-863e-678a337f87bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:11:12.024163 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:12.024110 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dea747ba-4708-4307-8744-911dea32712e-plugins-config-volume" pod "dea747ba-4708-4307-8744-911dea32712e" (UID: "dea747ba-4708-4307-8744-911dea32712e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:11:12.025848 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:12.025826 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dea747ba-4708-4307-8744-911dea32712e-kube-api-access-rj957" pod "dea747ba-4708-4307-8744-911dea32712e" (UID: "dea747ba-4708-4307-8744-911dea32712e"). InnerVolumeSpecName "kube-api-access-rj957". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:11:12.026344 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:12.026320 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b5b4db-cdc6-4d3d-863e-678a337f87bb-kube-api-access-qrlqw" pod "b6b5b4db-cdc6-4d3d-863e-678a337f87bb" (UID: "b6b5b4db-cdc6-4d3d-863e-678a337f87bb"). InnerVolumeSpecName "kube-api-access-qrlqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:11:12.124775 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:12.124751 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrlqw\" (UniqueName: \"kubernetes.io/projected/b6b5b4db-cdc6-4d3d-863e-678a337f87bb-kube-api-access-qrlqw\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:11:12.124775 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:12.124775 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dea747ba-4708-4307-8744-911dea32712e-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:11:12.124942 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:12.124785 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj957\" (UniqueName: \"kubernetes.io/projected/dea747ba-4708-4307-8744-911dea32712e-kube-api-access-rj957\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:11:12.124942 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:12.124793 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b5b4db-cdc6-4d3d-863e-678a337f87bb-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:11:12.322192 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:12.322116 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a23c98-787c476954-j5ls2"] Oct 30 18:11:12.326070 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:12.326042 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a23c98-787c476954-j5ls2"] Oct 30 18:11:12.387074 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:12.387045 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a23c98-76b6d5c87-8tx7q"] Oct 30 18:11:12.435142 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:12.435110 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a23c98-76b6d5c87-8tx7q"] Oct 30 18:11:14.096350 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:14.096320 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6b5b4db-cdc6-4d3d-863e-678a337f87bb" path="/var/lib/kubelet/pods/b6b5b4db-cdc6-4d3d-863e-678a337f87bb/volumes" Oct 30 18:11:14.096731 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:14.096687 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dea747ba-4708-4307-8744-911dea32712e" path="/var/lib/kubelet/pods/dea747ba-4708-4307-8744-911dea32712e/volumes" Oct 30 18:11:16.115348 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:16.115312 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" containerID="cri-o://e5d52c9f64ca2e613bacee99ea76eaf6ddc3c839441e7fbfea139a464eb2e052" gracePeriod=45 Oct 30 18:11:16.374565 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:16.374509 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 18:11:16.463891 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:16.463846 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13edf9da-7593-4b14-9830-f62cd466a6e4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13edf9da-7593-4b14-9830-f62cd466a6e4-tmp-dir\") pod \"13edf9da-7593-4b14-9830-f62cd466a6e4\" (UID: \"13edf9da-7593-4b14-9830-f62cd466a6e4\") " Oct 30 18:11:16.464099 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:16.463920 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13edf9da-7593-4b14-9830-f62cd466a6e4-tls-certs\" (UniqueName: \"kubernetes.io/secret/13edf9da-7593-4b14-9830-f62cd466a6e4-tls-certs\") pod \"13edf9da-7593-4b14-9830-f62cd466a6e4\" (UID: \"13edf9da-7593-4b14-9830-f62cd466a6e4\") " Oct 30 18:11:16.464099 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:16.463995 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13edf9da-7593-4b14-9830-f62cd466a6e4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/13edf9da-7593-4b14-9830-f62cd466a6e4-dshm\") pod \"13edf9da-7593-4b14-9830-f62cd466a6e4\" (UID: \"13edf9da-7593-4b14-9830-f62cd466a6e4\") " Oct 30 18:11:16.464244 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:16.464106 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13edf9da-7593-4b14-9830-f62cd466a6e4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13edf9da-7593-4b14-9830-f62cd466a6e4-model-cache\") pod \"13edf9da-7593-4b14-9830-f62cd466a6e4\" (UID: \"13edf9da-7593-4b14-9830-f62cd466a6e4\") " Oct 30 18:11:16.464244 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:16.464168 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13edf9da-7593-4b14-9830-f62cd466a6e4-kube-api-access-z7kqs\" (UniqueName: \"kubernetes.io/projected/13edf9da-7593-4b14-9830-f62cd466a6e4-kube-api-access-z7kqs\") pod \"13edf9da-7593-4b14-9830-f62cd466a6e4\" (UID: \"13edf9da-7593-4b14-9830-f62cd466a6e4\") " Oct 30 18:11:16.464244 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:16.464180 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13edf9da-7593-4b14-9830-f62cd466a6e4-tmp-dir" pod "13edf9da-7593-4b14-9830-f62cd466a6e4" (UID: "13edf9da-7593-4b14-9830-f62cd466a6e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:11:16.464244 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:16.464196 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13edf9da-7593-4b14-9830-f62cd466a6e4-home\" (UniqueName: \"kubernetes.io/empty-dir/13edf9da-7593-4b14-9830-f62cd466a6e4-home\") pod \"13edf9da-7593-4b14-9830-f62cd466a6e4\" (UID: \"13edf9da-7593-4b14-9830-f62cd466a6e4\") " Oct 30 18:11:16.464478 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:16.464362 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13edf9da-7593-4b14-9830-f62cd466a6e4-model-cache" pod "13edf9da-7593-4b14-9830-f62cd466a6e4" (UID: "13edf9da-7593-4b14-9830-f62cd466a6e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:11:16.464536 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:16.464497 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13edf9da-7593-4b14-9830-f62cd466a6e4-home" pod "13edf9da-7593-4b14-9830-f62cd466a6e4" (UID: "13edf9da-7593-4b14-9830-f62cd466a6e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:11:16.464536 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:16.464531 4965 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13edf9da-7593-4b14-9830-f62cd466a6e4-model-cache\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:11:16.464652 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:16.464548 4965 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13edf9da-7593-4b14-9830-f62cd466a6e4-tmp-dir\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:11:16.466331 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:16.466301 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13edf9da-7593-4b14-9830-f62cd466a6e4-dshm" pod "13edf9da-7593-4b14-9830-f62cd466a6e4" (UID: "13edf9da-7593-4b14-9830-f62cd466a6e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:11:16.466331 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:16.466317 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13edf9da-7593-4b14-9830-f62cd466a6e4-tls-certs" pod "13edf9da-7593-4b14-9830-f62cd466a6e4" (UID: "13edf9da-7593-4b14-9830-f62cd466a6e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:11:16.466557 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:16.466539 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13edf9da-7593-4b14-9830-f62cd466a6e4-kube-api-access-z7kqs" pod "13edf9da-7593-4b14-9830-f62cd466a6e4" (UID: "13edf9da-7593-4b14-9830-f62cd466a6e4"). InnerVolumeSpecName "kube-api-access-z7kqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:11:16.565899 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:16.565825 4965 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13edf9da-7593-4b14-9830-f62cd466a6e4-dshm\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:11:16.565899 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:16.565894 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7kqs\" (UniqueName: \"kubernetes.io/projected/13edf9da-7593-4b14-9830-f62cd466a6e4-kube-api-access-z7kqs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:11:16.566126 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:16.565912 4965 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13edf9da-7593-4b14-9830-f62cd466a6e4-home\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:11:16.566126 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:16.565924 4965 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13edf9da-7593-4b14-9830-f62cd466a6e4-tls-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:11:17.012357 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:17.012324 4965 generic.go:356] "Generic (PLEG): container finished" podID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerID="e5d52c9f64ca2e613bacee99ea76eaf6ddc3c839441e7fbfea139a464eb2e052" exitCode=0 Oct 30 18:11:17.012545 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:17.012368 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" Oct 30 18:11:17.012545 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:17.012397 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" event={"ID":"13edf9da-7593-4b14-9830-f62cd466a6e4","Type":"ContainerDied","Data":"e5d52c9f64ca2e613bacee99ea76eaf6ddc3c839441e7fbfea139a464eb2e052"} Oct 30 18:11:17.012545 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:17.012427 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n" event={"ID":"13edf9da-7593-4b14-9830-f62cd466a6e4","Type":"ContainerDied","Data":"8ef70040331b11bdd90e60e6f59f8ffe589034d0b05f74a179e6b8e4c47f23a0"} Oct 30 18:11:17.012545 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:17.012443 4965 scope.go:122] "RemoveContainer" containerID="e5d52c9f64ca2e613bacee99ea76eaf6ddc3c839441e7fbfea139a464eb2e052" Oct 30 18:11:17.022256 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:17.022239 4965 scope.go:122] "RemoveContainer" containerID="e5d52c9f64ca2e613bacee99ea76eaf6ddc3c839441e7fbfea139a464eb2e052" Oct 30 18:11:17.022510 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:11:17.022491 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5d52c9f64ca2e613bacee99ea76eaf6ddc3c839441e7fbfea139a464eb2e052\": container with ID starting with e5d52c9f64ca2e613bacee99ea76eaf6ddc3c839441e7fbfea139a464eb2e052 not found: ID does not exist" containerID="e5d52c9f64ca2e613bacee99ea76eaf6ddc3c839441e7fbfea139a464eb2e052" Oct 30 18:11:17.022560 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:17.022517 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5d52c9f64ca2e613bacee99ea76eaf6ddc3c839441e7fbfea139a464eb2e052"} err="failed to get container status \"e5d52c9f64ca2e613bacee99ea76eaf6ddc3c839441e7fbfea139a464eb2e052\": rpc error: code = NotFound desc = could not find container \"e5d52c9f64ca2e613bacee99ea76eaf6ddc3c839441e7fbfea139a464eb2e052\": container with ID starting with e5d52c9f64ca2e613bacee99ea76eaf6ddc3c839441e7fbfea139a464eb2e052 not found: ID does not exist" Oct 30 18:11:17.062568 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:17.062543 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n"] Oct 30 18:11:17.074831 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:17.074800 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0e7403/simulator-w4-0e7403-kserve-5668b8f548-5455n"] Oct 30 18:11:18.096973 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:18.096940 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" path="/var/lib/kubelet/pods/13edf9da-7593-4b14-9830-f62cd466a6e4/volumes" Oct 30 18:11:20.244363 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:20.244330 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:11:26.996999 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:26.996964 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:11:35.995904 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:11:35.995798 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:11:36.030334 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:36.030296 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1be6b995-n4hfb"] Oct 30 18:11:36.042456 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:36.042427 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1be6b995-n4hfb"] Oct 30 18:11:36.098952 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:11:36.098916 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b751169-3514-4c01-9b26-986228c48aab" path="/var/lib/kubelet/pods/0b751169-3514-4c01-9b26-986228c48aab/volumes" Oct 30 18:12:05.231904 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.231848 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25"] Oct 30 18:12:05.232397 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.232306 4965 cpu_manager.go:414] "containerMap: removing container" podUID="ef413002-b509-4658-a123-a5b4b0715d58" containerName="istio-proxy" Oct 30 18:12:05.232397 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.232322 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef413002-b509-4658-a123-a5b4b0715d58" containerName="istio-proxy" Oct 30 18:12:05.232397 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.232350 4965 cpu_manager.go:414] "containerMap: removing container" podUID="b6b5b4db-cdc6-4d3d-863e-678a337f87bb" containerName="payload-processing" Oct 30 18:12:05.232397 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.232355 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b5b4db-cdc6-4d3d-863e-678a337f87bb" containerName="payload-processing" Oct 30 18:12:05.232397 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.232363 4965 cpu_manager.go:414] "containerMap: removing container" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" Oct 30 18:12:05.232397 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.232368 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" Oct 30 18:12:05.232397 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.232380 4965 cpu_manager.go:414] "containerMap: removing container" podUID="065dad47-8b1a-4b1c-87df-80e3f106e54c" containerName="revoke-keys" Oct 30 18:12:05.232397 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.232385 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="065dad47-8b1a-4b1c-87df-80e3f106e54c" containerName="revoke-keys" Oct 30 18:12:05.232397 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.232391 4965 cpu_manager.go:414] "containerMap: removing container" podUID="dea747ba-4708-4307-8744-911dea32712e" containerName="payload-pre-processing" Oct 30 18:12:05.232397 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.232397 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dea747ba-4708-4307-8744-911dea32712e" containerName="payload-pre-processing" Oct 30 18:12:05.232729 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.232462 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b5b4db-cdc6-4d3d-863e-678a337f87bb" containerName="payload-processing" Oct 30 18:12:05.232729 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.232468 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="13edf9da-7593-4b14-9830-f62cd466a6e4" containerName="main" Oct 30 18:12:05.232729 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.232476 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="065dad47-8b1a-4b1c-87df-80e3f106e54c" containerName="revoke-keys" Oct 30 18:12:05.232729 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.232481 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="dea747ba-4708-4307-8744-911dea32712e" containerName="payload-pre-processing" Oct 30 18:12:05.232729 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.232489 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef413002-b509-4658-a123-a5b4b0715d58" containerName="istio-proxy" Oct 30 18:12:05.235884 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.235850 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:05.265456 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.265423 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25"] Oct 30 18:12:05.314713 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.314679 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-rkm25\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:05.314858 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.314716 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e0d78c2-d701-4ecc-888c-d69469f844ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-rkm25\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:05.314858 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.314734 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e0d78c2-d701-4ecc-888c-d69469f844ee-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-rkm25\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:05.314858 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.314818 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tx5q\" (UniqueName: \"kubernetes.io/projected/9e0d78c2-d701-4ecc-888c-d69469f844ee-kube-api-access-5tx5q\") pod \"maas-default-gateway-openshift-default-7df854bf67-rkm25\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:05.315029 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.314907 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-rkm25\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:05.315029 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.314942 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-rkm25\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:05.315029 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.314967 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-rkm25\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:05.315029 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.314996 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e0d78c2-d701-4ecc-888c-d69469f844ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-rkm25\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:05.315167 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.315100 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-rkm25\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:05.416151 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.416121 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-rkm25\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:05.416151 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.416155 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e0d78c2-d701-4ecc-888c-d69469f844ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-rkm25\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:05.416388 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.416172 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e0d78c2-d701-4ecc-888c-d69469f844ee-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-rkm25\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:05.416388 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.416206 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tx5q\" (UniqueName: \"kubernetes.io/projected/9e0d78c2-d701-4ecc-888c-d69469f844ee-kube-api-access-5tx5q\") pod \"maas-default-gateway-openshift-default-7df854bf67-rkm25\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:05.416388 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.416257 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-rkm25\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:05.416388 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.416286 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-rkm25\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:05.416388 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.416317 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-rkm25\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:05.416388 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.416356 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e0d78c2-d701-4ecc-888c-d69469f844ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-rkm25\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:05.416734 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.416420 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-rkm25\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:05.416734 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.416582 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-rkm25\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:05.416889 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.416728 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-rkm25\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:05.416889 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.416793 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-rkm25\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:05.416889 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.416875 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-rkm25\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:05.417024 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.416956 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e0d78c2-d701-4ecc-888c-d69469f844ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-rkm25\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:05.418567 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.418539 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-rkm25\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:05.419012 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.418990 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e0d78c2-d701-4ecc-888c-d69469f844ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-rkm25\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:05.439298 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.439271 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e0d78c2-d701-4ecc-888c-d69469f844ee-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-rkm25\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:05.439515 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.439497 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tx5q\" (UniqueName: \"kubernetes.io/projected/9e0d78c2-d701-4ecc-888c-d69469f844ee-kube-api-access-5tx5q\") pod \"maas-default-gateway-openshift-default-7df854bf67-rkm25\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:05.549024 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.548930 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:05.863683 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.863657 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25"] Oct 30 18:12:05.865088 ip-10-0-139-222 kubenswrapper[4965]: W1030 18:12:05.865065 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e0d78c2_d701_4ecc_888c_d69469f844ee.slice/crio-91e73e3a005bbf4f4de99c841f0354ed55dc03b6e0bd9fa274c17b64d161cb84 WatchSource:0}: Error finding container 91e73e3a005bbf4f4de99c841f0354ed55dc03b6e0bd9fa274c17b64d161cb84: Status 404 returned error can't find the container with id 91e73e3a005bbf4f4de99c841f0354ed55dc03b6e0bd9fa274c17b64d161cb84 Oct 30 18:12:05.867224 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.867193 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 18:12:05.867307 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.867260 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 18:12:05.867307 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:05.867290 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 18:12:05.996986 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:12:05.996947 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:12:06.262257 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:06.262168 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" event={"ID":"9e0d78c2-d701-4ecc-888c-d69469f844ee","Type":"ContainerStarted","Data":"7b5c84018529e842a8e1b51e172f6627918577b548c8171bd207f7acd16f7f1c"} Oct 30 18:12:06.262257 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:06.262224 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" event={"ID":"9e0d78c2-d701-4ecc-888c-d69469f844ee","Type":"ContainerStarted","Data":"91e73e3a005bbf4f4de99c841f0354ed55dc03b6e0bd9fa274c17b64d161cb84"} Oct 30 18:12:06.313824 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:06.313767 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" podStartSLOduration=1.313751897 podStartE2EDuration="1.313751897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:06.311933698 +0000 UTC m=+3584.364033626" watchObservedRunningTime="2026-10-30 18:12:06.313751897 +0000 UTC m=+3584.365851820" Oct 30 18:12:06.549485 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:06.549399 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:06.551250 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:06.551221 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rkm25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 18:12:06.551390 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:06.551270 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" podUID="9e0d78c2-d701-4ecc-888c-d69469f844ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 18:12:07.047906 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:07.047874 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d439241-mf8d2"] Oct 30 18:12:07.064889 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:07.063407 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d439241-mf8d2"] Oct 30 18:12:07.549978 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:07.549942 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rkm25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 18:12:07.550404 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:07.550004 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" podUID="9e0d78c2-d701-4ecc-888c-d69469f844ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 18:12:08.097057 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:08.097022 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c390c71-420f-489f-9a84-0ac10e6a74ef" path="/var/lib/kubelet/pods/8c390c71-420f-489f-9a84-0ac10e6a74ef/volumes" Oct 30 18:12:08.549706 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:08.549609 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rkm25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 18:12:08.549706 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:08.549671 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" podUID="9e0d78c2-d701-4ecc-888c-d69469f844ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 18:12:09.550116 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:09.550077 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rkm25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 18:12:09.550535 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:09.550148 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" podUID="9e0d78c2-d701-4ecc-888c-d69469f844ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 18:12:10.549562 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:10.549526 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rkm25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 18:12:10.549765 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:10.549588 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" podUID="9e0d78c2-d701-4ecc-888c-d69469f844ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 18:12:11.550117 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:11.550076 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rkm25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 18:12:11.550557 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:11.550134 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" podUID="9e0d78c2-d701-4ecc-888c-d69469f844ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 18:12:12.550037 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:12.550002 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rkm25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 18:12:12.550259 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:12.550055 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" podUID="9e0d78c2-d701-4ecc-888c-d69469f844ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 18:12:14.125265 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:14.125232 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:14.125739 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:14.125625 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:14.126445 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:14.126421 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:12:22.910153 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:22.910021 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 18:12:22.911787 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:22.911763 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 18:12:35.996078 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:12:35.996030 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:12:52.938072 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:12:52.938034 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:13:05.424454 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:05.424373 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq"] Oct 30 18:13:05.428059 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:05.428040 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:05.459552 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:05.459524 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq"] Oct 30 18:13:05.513116 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:05.513066 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-pvlkq\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:05.513283 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:05.513147 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2zr6\" (UniqueName: \"kubernetes.io/projected/0c1403e9-4637-49cc-9e50-f7be9caa1847-kube-api-access-j2zr6\") pod \"maas-default-gateway-openshift-default-7df854bf67-pvlkq\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:05.513283 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:05.513229 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-pvlkq\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:05.513415 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:05.513298 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-pvlkq\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:05.513415 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:05.513351 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-pvlkq\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:05.513531 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:05.513423 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c1403e9-4637-49cc-9e50-f7be9caa1847-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-pvlkq\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:05.513531 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:05.513456 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c1403e9-4637-49cc-9e50-f7be9caa1847-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-pvlkq\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:05.513531 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:05.513487 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-pvlkq\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:05.513705 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:05.513536 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c1403e9-4637-49cc-9e50-f7be9caa1847-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-pvlkq\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:05.614775 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:05.614742 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-pvlkq\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:05.614970 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:05.614793 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2zr6\" (UniqueName: \"kubernetes.io/projected/0c1403e9-4637-49cc-9e50-f7be9caa1847-kube-api-access-j2zr6\") pod \"maas-default-gateway-openshift-default-7df854bf67-pvlkq\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:05.614970 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:05.614810 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-pvlkq\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:05.614970 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:05.614838 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-pvlkq\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:05.614970 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:05.614854 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-pvlkq\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:05.614970 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:05.614903 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c1403e9-4637-49cc-9e50-f7be9caa1847-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-pvlkq\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:05.614970 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:05.614931 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c1403e9-4637-49cc-9e50-f7be9caa1847-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-pvlkq\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:05.614970 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:05.614961 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-pvlkq\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:05.615366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:05.615012 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c1403e9-4637-49cc-9e50-f7be9caa1847-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-pvlkq\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:05.615366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:05.615243 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-pvlkq\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:05.615366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:05.615288 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-pvlkq\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:05.615366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:05.615346 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-pvlkq\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:05.615539 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:05.615485 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-pvlkq\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:05.615704 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:05.615682 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c1403e9-4637-49cc-9e50-f7be9caa1847-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-pvlkq\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:05.617391 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:05.617366 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-pvlkq\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:05.617599 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:05.617582 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c1403e9-4637-49cc-9e50-f7be9caa1847-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-pvlkq\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:05.638503 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:05.638480 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c1403e9-4637-49cc-9e50-f7be9caa1847-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-pvlkq\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:05.638717 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:05.638694 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2zr6\" (UniqueName: \"kubernetes.io/projected/0c1403e9-4637-49cc-9e50-f7be9caa1847-kube-api-access-j2zr6\") pod \"maas-default-gateway-openshift-default-7df854bf67-pvlkq\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:05.741079 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:05.740997 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:05.997029 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:13:05.996941 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:13:06.063119 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:06.063092 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq"] Oct 30 18:13:06.064978 ip-10-0-139-222 kubenswrapper[4965]: W1030 18:13:06.064942 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c1403e9_4637_49cc_9e50_f7be9caa1847.slice/crio-85fa8b4017d7a2f9e4771340ec641c7f251780d117e9b0bb3af643022efa9269 WatchSource:0}: Error finding container 85fa8b4017d7a2f9e4771340ec641c7f251780d117e9b0bb3af643022efa9269: Status 404 returned error can't find the container with id 85fa8b4017d7a2f9e4771340ec641c7f251780d117e9b0bb3af643022efa9269 Oct 30 18:13:06.067297 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:06.067266 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 18:13:06.067378 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:06.067335 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 18:13:06.067418 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:06.067382 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 18:13:06.588526 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:06.588492 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" event={"ID":"0c1403e9-4637-49cc-9e50-f7be9caa1847","Type":"ContainerStarted","Data":"565c35901d80889fffe733d06e2114cd7ea4709590807a38a5d8c5e68d730c01"} Oct 30 18:13:06.588977 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:06.588541 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" event={"ID":"0c1403e9-4637-49cc-9e50-f7be9caa1847","Type":"ContainerStarted","Data":"85fa8b4017d7a2f9e4771340ec641c7f251780d117e9b0bb3af643022efa9269"} Oct 30 18:13:06.678887 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:06.678550 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" podStartSLOduration=1.67852819 podStartE2EDuration="1.67852819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:06.64498349 +0000 UTC m=+3644.697083440" watchObservedRunningTime="2026-10-30 18:13:06.67852819 +0000 UTC m=+3644.730628122" Oct 30 18:13:06.742136 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:06.742100 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:07.742733 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:07.742700 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-pvlkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 18:13:07.743152 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:07.742754 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" podUID="0c1403e9-4637-49cc-9e50-f7be9caa1847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 18:13:08.742437 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:08.742396 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-pvlkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 18:13:08.742635 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:08.742462 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" podUID="0c1403e9-4637-49cc-9e50-f7be9caa1847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 18:13:09.741886 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:09.741822 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-pvlkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 18:13:09.742338 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:09.741902 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" podUID="0c1403e9-4637-49cc-9e50-f7be9caa1847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 18:13:10.420137 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:10.420092 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:13:10.741683 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:10.741604 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-pvlkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 18:13:10.741683 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:10.741657 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" podUID="0c1403e9-4637-49cc-9e50-f7be9caa1847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 18:13:11.742057 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:11.742016 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-pvlkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 18:13:11.742565 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:11.742092 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" podUID="0c1403e9-4637-49cc-9e50-f7be9caa1847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 18:13:12.741644 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:12.741612 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-pvlkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 18:13:12.741826 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:12.741663 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" podUID="0c1403e9-4637-49cc-9e50-f7be9caa1847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 18:13:14.742805 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:14.742756 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-pvlkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:13:14.743299 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:14.742831 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" podUID="0c1403e9-4637-49cc-9e50-f7be9caa1847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:15.784107 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:15.784056 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-pvlkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:13:15.784617 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:15.784136 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" podUID="0c1403e9-4637-49cc-9e50-f7be9caa1847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:16.392219 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:16.392186 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:16.392810 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:16.392785 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:16.393733 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:16.393706 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:13:23.549991 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:23.549958 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:13:27.239689 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:27.239657 4965 scope.go:122] "RemoveContainer" containerID="17f480667741ef51803e733019ef1de6b2bca823b4002199418f1c1f460567b3" Oct 30 18:13:27.251893 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:27.251850 4965 scope.go:122] "RemoveContainer" containerID="6a2b87a90e8346c823b44d6d5a0b80de37f87fa790f9874c4ae47e583b173125" Oct 30 18:13:35.996651 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:13:35.996613 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:13:41.049663 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:41.049624 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:13:46.030075 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:46.030035 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c6d3a2c-wkpzc"] Oct 30 18:13:46.040799 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:46.040770 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c6d3a2c-wkpzc"] Oct 30 18:13:46.139980 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:46.139940 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93b77541-e76a-4b77-99f7-68e1e5e2a624" path="/var/lib/kubelet/pods/93b77541-e76a-4b77-99f7-68e1e5e2a624/volumes" Oct 30 18:13:56.518390 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:13:56.518351 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:14:05.994589 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:14:05.994544 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:14:30.218326 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:14:30.218290 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:14:35.996281 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:14:35.996201 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:14:40.620690 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:14:40.620654 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:15:04.704366 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:15:04.704332 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:15:06.047095 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:15:06.047056 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:15:22.231730 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:15:22.231692 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:15:27.289887 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:15:27.285414 4965 scope.go:122] "RemoveContainer" containerID="0c41b809b9a6d3dd6b5f8b476e76d77514baba6a7a196aaafdc2a5edbea24728" Oct 30 18:15:27.336583 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:15:27.336553 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:15:35.995907 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:15:35.995856 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:15:43.825221 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:15:43.825180 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:15:55.919314 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:15:55.919280 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:16:05.995422 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:16:05.995339 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:16:13.530657 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:16:13.530619 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:16:30.720741 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:16:30.720707 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:16:35.996059 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:16:35.996013 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:16:41.432767 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:16:41.432731 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:16:46.923553 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:16:46.923515 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:17:03.728574 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:03.728534 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:17:05.995450 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:17:05.995412 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:17:08.828322 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:08.828285 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:17:22.944902 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:22.944875 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 18:17:22.945385 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:22.944875 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 18:17:27.350342 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:27.350310 4965 scope.go:122] "RemoveContainer" containerID="7dcc2c0941e016a529075d13dc131314a4014699394b228a6df7cd0ae08de9b6" Oct 30 18:17:27.502203 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:27.502173 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:17:32.518889 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:32.518842 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:17:35.996056 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:17:35.995966 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:17:41.239464 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.239429 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb"] Oct 30 18:17:41.244182 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.244153 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:41.260652 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.260622 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-05e969-openshift-default-dockercfg-cv797\"" Oct 30 18:17:41.275778 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.275744 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb"] Oct 30 18:17:41.290116 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.290089 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istio-token\") pod \"e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:41.290265 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.290138 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istiod-ca-cert\") pod \"e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:41.290265 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.290238 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istio-envoy\") pod \"e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:41.290400 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.290261 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istio-data\") pod \"e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:41.290400 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.290337 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrllj\" (UniqueName: \"kubernetes.io/projected/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-kube-api-access-zrllj\") pod \"e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:41.290521 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.290449 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-workload-socket\") pod \"e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:41.290521 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.290491 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-workload-certs\") pod \"e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:41.290644 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.290577 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istio-podinfo\") pod \"e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:41.290644 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.290619 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-credential-socket\") pod \"e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:41.391479 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.391439 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istio-data\") pod \"e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:41.391694 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.391490 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrllj\" (UniqueName: \"kubernetes.io/projected/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-kube-api-access-zrllj\") pod \"e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:41.391694 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.391552 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-workload-socket\") pod \"e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:41.391694 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.391584 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-workload-certs\") pod \"e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:41.391694 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.391629 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istio-podinfo\") pod \"e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:41.391694 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.391659 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-credential-socket\") pod \"e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:41.391694 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.391684 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istio-token\") pod \"e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:41.392084 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.391731 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istiod-ca-cert\") pod \"e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:41.392084 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.391768 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istio-envoy\") pod \"e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:41.392084 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.392000 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istio-data\") pod \"e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:41.392331 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.392309 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-workload-socket\") pod \"e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:41.392602 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.392580 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-credential-socket\") pod \"e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:41.393081 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.393021 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istiod-ca-cert\") pod \"e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:41.393602 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.393582 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-workload-certs\") pod \"e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:41.404194 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.404163 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istio-envoy\") pod \"e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:41.404352 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.404331 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istio-podinfo\") pod \"e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:41.415136 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.415085 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istio-token\") pod \"e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:41.415807 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.415776 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrllj\" (UniqueName: \"kubernetes.io/projected/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-kube-api-access-zrllj\") pod \"e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:41.559154 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.559075 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:41.887224 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.887194 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb"] Oct 30 18:17:41.888904 ip-10-0-139-222 kubenswrapper[4965]: W1030 18:17:41.888843 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a8f0904_a1a9_48fe_a499_5ec1f5d141c6.slice/crio-789e70b6c90e9b6f0b41c9b4c836336be2b76cb32b44c2dce7461c4c390ffcc9 WatchSource:0}: Error finding container 789e70b6c90e9b6f0b41c9b4c836336be2b76cb32b44c2dce7461c4c390ffcc9: Status 404 returned error can't find the container with id 789e70b6c90e9b6f0b41c9b4c836336be2b76cb32b44c2dce7461c4c390ffcc9 Oct 30 18:17:41.891033 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.890998 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 18:17:41.891123 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.891065 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 18:17:41.891123 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:41.891095 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 18:17:42.101492 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:42.101457 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" event={"ID":"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6","Type":"ContainerStarted","Data":"eb920058f88fa5a4c970cd4fdeb43e892437ad6160387f60ec62ddc9f6e39778"} Oct 30 18:17:42.101698 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:42.101509 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" event={"ID":"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6","Type":"ContainerStarted","Data":"789e70b6c90e9b6f0b41c9b4c836336be2b76cb32b44c2dce7461c4c390ffcc9"} Oct 30 18:17:42.147125 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:42.147031 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" podStartSLOduration=1.147018226 podStartE2EDuration="1.147018226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:42.145921452 +0000 UTC m=+3920.198021385" watchObservedRunningTime="2026-10-30 18:17:42.147018226 +0000 UTC m=+3920.199118157" Oct 30 18:17:42.560169 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:42.560073 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:42.565455 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:42.565428 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:43.107401 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:43.107370 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:43.108406 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:43.108379 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:17:46.384898 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:46.384856 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-05e969-c8cf69488k2sdl"] Oct 30 18:17:46.388695 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:46.388677 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-05e969-c8cf69488k2sdl" Oct 30 18:17:46.407962 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:46.407940 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-05e969-c8cf69488k2sdl"] Oct 30 18:17:46.417248 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:46.417228 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-f5f089c9-dockercfg-vmbgz\"" Oct 30 18:17:46.417372 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:46.417258 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-05e969\"" Oct 30 18:17:46.431689 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:46.431663 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e722b15b-5e82-47e5-b5f6-34d411f8f8fc-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-05e969-c8cf69488k2sdl\" (UID: \"e722b15b-5e82-47e5-b5f6-34d411f8f8fc\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-05e969-c8cf69488k2sdl" Oct 30 18:17:46.431805 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:46.431744 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-557x6\" (UniqueName: \"kubernetes.io/projected/e722b15b-5e82-47e5-b5f6-34d411f8f8fc-kube-api-access-557x6\") pod \"payload-processing-e2e-worker-single-main-05e969-c8cf69488k2sdl\" (UID: \"e722b15b-5e82-47e5-b5f6-34d411f8f8fc\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-05e969-c8cf69488k2sdl" Oct 30 18:17:46.534642 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:46.534603 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e722b15b-5e82-47e5-b5f6-34d411f8f8fc-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-05e969-c8cf69488k2sdl\" (UID: \"e722b15b-5e82-47e5-b5f6-34d411f8f8fc\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-05e969-c8cf69488k2sdl" Oct 30 18:17:46.534845 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:46.534686 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-557x6\" (UniqueName: \"kubernetes.io/projected/e722b15b-5e82-47e5-b5f6-34d411f8f8fc-kube-api-access-557x6\") pod \"payload-processing-e2e-worker-single-main-05e969-c8cf69488k2sdl\" (UID: \"e722b15b-5e82-47e5-b5f6-34d411f8f8fc\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-05e969-c8cf69488k2sdl" Oct 30 18:17:46.536138 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:46.536105 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e722b15b-5e82-47e5-b5f6-34d411f8f8fc-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-05e969-c8cf69488k2sdl\" (UID: \"e722b15b-5e82-47e5-b5f6-34d411f8f8fc\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-05e969-c8cf69488k2sdl" Oct 30 18:17:46.557847 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:46.557814 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-557x6\" (UniqueName: \"kubernetes.io/projected/e722b15b-5e82-47e5-b5f6-34d411f8f8fc-kube-api-access-557x6\") pod \"payload-processing-e2e-worker-single-main-05e969-c8cf69488k2sdl\" (UID: \"e722b15b-5e82-47e5-b5f6-34d411f8f8fc\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-05e969-c8cf69488k2sdl" Oct 30 18:17:46.703085 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:46.703000 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-05e969-c8cf69488k2sdl" Oct 30 18:17:47.005135 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:47.004345 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-05e969-c8cf69488k2sdl"] Oct 30 18:17:47.008297 ip-10-0-139-222 kubenswrapper[4965]: W1030 18:17:47.008261 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode722b15b_5e82_47e5_b5f6_34d411f8f8fc.slice/crio-45a6b83ce041cd154969ae9166c4a580e3747865ec5e0edc6ee23cd9bb7b3da2 WatchSource:0}: Error finding container 45a6b83ce041cd154969ae9166c4a580e3747865ec5e0edc6ee23cd9bb7b3da2: Status 404 returned error can't find the container with id 45a6b83ce041cd154969ae9166c4a580e3747865ec5e0edc6ee23cd9bb7b3da2 Oct 30 18:17:47.130950 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:47.130845 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-05e969-c8cf69488k2sdl" event={"ID":"e722b15b-5e82-47e5-b5f6-34d411f8f8fc","Type":"ContainerStarted","Data":"dc1cc5e2cda5da3d3fdbe6e4d75fd13db8caf6e4b3750640d34ed3d4f288ecc6"} Oct 30 18:17:47.130950 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:47.130921 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-05e969-c8cf69488k2sdl" event={"ID":"e722b15b-5e82-47e5-b5f6-34d411f8f8fc","Type":"ContainerStarted","Data":"45a6b83ce041cd154969ae9166c4a580e3747865ec5e0edc6ee23cd9bb7b3da2"} Oct 30 18:17:47.131169 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:47.131066 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-05e969-c8cf69488k2sdl" Oct 30 18:17:47.175562 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:47.175511 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-05e969-c8cf69488k2sdl" podStartSLOduration=1.175497658 podStartE2EDuration="1.175497658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:47.174518199 +0000 UTC m=+3925.226618152" watchObservedRunningTime="2026-10-30 18:17:47.175497658 +0000 UTC m=+3925.227597581" Oct 30 18:17:49.921407 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:49.921367 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:17:58.138496 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:58.138471 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-05e969-c8cf69488k2sdl" Oct 30 18:17:59.150012 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:59.149977 4965 patch_prober.go:29] interesting pod/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:17:59.150430 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:17:59.150033 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" podUID="2a8f0904-a1a9-48fe-a499-5ec1f5d141c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:00.221682 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:18:00.221644 4965 generic.go:356] "Generic (PLEG): container finished" podID="2a8f0904-a1a9-48fe-a499-5ec1f5d141c6" containerID="eb920058f88fa5a4c970cd4fdeb43e892437ad6160387f60ec62ddc9f6e39778" exitCode=0 Oct 30 18:18:00.222150 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:18:00.221715 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" event={"ID":"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6","Type":"ContainerDied","Data":"eb920058f88fa5a4c970cd4fdeb43e892437ad6160387f60ec62ddc9f6e39778"} Oct 30 18:18:00.222150 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:18:00.222116 4965 scope.go:122] "RemoveContainer" containerID="eb920058f88fa5a4c970cd4fdeb43e892437ad6160387f60ec62ddc9f6e39778" Oct 30 18:18:00.223850 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:18:00.223796 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 18:18:00.224188 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:18:00.224157 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 18:18:00.224389 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:18:00.224366 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 18:18:01.228967 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:18:01.228932 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" event={"ID":"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6","Type":"ContainerStarted","Data":"557c6d6c688f3babfa878385b06ca300b80077a0dcfc4c405cbf3b00773b6b1a"} Oct 30 18:18:01.577952 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:18:01.574650 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:18:01.577952 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:18:01.574796 4965 patch_prober.go:29] interesting pod/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 18:18:01.577952 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:18:01.574856 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" podUID="2a8f0904-a1a9-48fe-a499-5ec1f5d141c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 18:18:02.578886 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:18:02.571341 4965 patch_prober.go:29] interesting pod/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 18:18:02.578886 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:18:02.571402 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" podUID="2a8f0904-a1a9-48fe-a499-5ec1f5d141c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 18:18:03.563888 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:18:03.560258 4965 patch_prober.go:29] interesting pod/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 18:18:03.563888 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:18:03.560322 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" podUID="2a8f0904-a1a9-48fe-a499-5ec1f5d141c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 18:18:04.571885 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:18:04.564283 4965 patch_prober.go:29] interesting pod/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 18:18:04.571885 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:18:04.564351 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" podUID="2a8f0904-a1a9-48fe-a499-5ec1f5d141c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 18:18:05.562361 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:18:05.562319 4965 patch_prober.go:29] interesting pod/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 18:18:05.562673 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:18:05.562645 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" podUID="2a8f0904-a1a9-48fe-a499-5ec1f5d141c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 18:18:06.007041 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:18:06.007012 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:18:06.144350 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:18:06.144317 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:18:07.014132 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:18:07.014099 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:18:07.014623 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:18:07.014385 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:18:07.015276 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:18:07.015254 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:18:19.602906 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:18:19.602873 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:18:35.994811 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:18:35.994773 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:18:37.029827 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:18:37.029791 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:18:47.294420 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:18:47.294383 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-05e969-c8cf69488k2sdl"] Oct 30 18:18:51.527537 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:18:51.527496 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:18:52.294972 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:18:52.294932 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-05e969-c8cf69488k2sdl" podUID="e722b15b-5e82-47e5-b5f6-34d411f8f8fc" containerName="payload-processing" containerID="cri-o://dc1cc5e2cda5da3d3fdbe6e4d75fd13db8caf6e4b3750640d34ed3d4f288ecc6" gracePeriod=25 Oct 30 18:18:56.283257 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:18:56.283221 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb"] Oct 30 18:18:56.283734 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:18:56.283492 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" podUID="2a8f0904-a1a9-48fe-a499-5ec1f5d141c6" containerName="istio-proxy" containerID="cri-o://557c6d6c688f3babfa878385b06ca300b80077a0dcfc4c405cbf3b00773b6b1a" gracePeriod=30 Oct 30 18:18:58.137096 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:18:58.137056 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-05e969-c8cf69488k2sdl" podUID="e722b15b-5e82-47e5-b5f6-34d411f8f8fc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:19:01.539468 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.539437 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:19:01.550904 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.550823 4965 generic.go:356] "Generic (PLEG): container finished" podID="2a8f0904-a1a9-48fe-a499-5ec1f5d141c6" containerID="557c6d6c688f3babfa878385b06ca300b80077a0dcfc4c405cbf3b00773b6b1a" exitCode=0 Oct 30 18:19:01.551051 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.550902 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" Oct 30 18:19:01.551051 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.550912 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" event={"ID":"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6","Type":"ContainerDied","Data":"557c6d6c688f3babfa878385b06ca300b80077a0dcfc4c405cbf3b00773b6b1a"} Oct 30 18:19:01.551051 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.550964 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb" event={"ID":"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6","Type":"ContainerDied","Data":"789e70b6c90e9b6f0b41c9b4c836336be2b76cb32b44c2dce7461c4c390ffcc9"} Oct 30 18:19:01.551051 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.550994 4965 scope.go:122] "RemoveContainer" containerID="557c6d6c688f3babfa878385b06ca300b80077a0dcfc4c405cbf3b00773b6b1a" Oct 30 18:19:01.559636 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.559613 4965 scope.go:122] "RemoveContainer" containerID="eb920058f88fa5a4c970cd4fdeb43e892437ad6160387f60ec62ddc9f6e39778" Oct 30 18:19:01.567615 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.567599 4965 scope.go:122] "RemoveContainer" containerID="557c6d6c688f3babfa878385b06ca300b80077a0dcfc4c405cbf3b00773b6b1a" Oct 30 18:19:01.567905 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:19:01.567882 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"557c6d6c688f3babfa878385b06ca300b80077a0dcfc4c405cbf3b00773b6b1a\": container with ID starting with 557c6d6c688f3babfa878385b06ca300b80077a0dcfc4c405cbf3b00773b6b1a not found: ID does not exist" containerID="557c6d6c688f3babfa878385b06ca300b80077a0dcfc4c405cbf3b00773b6b1a" Oct 30 18:19:01.567984 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.567913 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"557c6d6c688f3babfa878385b06ca300b80077a0dcfc4c405cbf3b00773b6b1a"} err="failed to get container status \"557c6d6c688f3babfa878385b06ca300b80077a0dcfc4c405cbf3b00773b6b1a\": rpc error: code = NotFound desc = could not find container \"557c6d6c688f3babfa878385b06ca300b80077a0dcfc4c405cbf3b00773b6b1a\": container with ID starting with 557c6d6c688f3babfa878385b06ca300b80077a0dcfc4c405cbf3b00773b6b1a not found: ID does not exist" Oct 30 18:19:01.567984 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.567929 4965 scope.go:122] "RemoveContainer" containerID="eb920058f88fa5a4c970cd4fdeb43e892437ad6160387f60ec62ddc9f6e39778" Oct 30 18:19:01.568140 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:19:01.568126 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb920058f88fa5a4c970cd4fdeb43e892437ad6160387f60ec62ddc9f6e39778\": container with ID starting with eb920058f88fa5a4c970cd4fdeb43e892437ad6160387f60ec62ddc9f6e39778 not found: ID does not exist" containerID="eb920058f88fa5a4c970cd4fdeb43e892437ad6160387f60ec62ddc9f6e39778" Oct 30 18:19:01.568186 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.568142 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb920058f88fa5a4c970cd4fdeb43e892437ad6160387f60ec62ddc9f6e39778"} err="failed to get container status \"eb920058f88fa5a4c970cd4fdeb43e892437ad6160387f60ec62ddc9f6e39778\": rpc error: code = NotFound desc = could not find container \"eb920058f88fa5a4c970cd4fdeb43e892437ad6160387f60ec62ddc9f6e39778\": container with ID starting with eb920058f88fa5a4c970cd4fdeb43e892437ad6160387f60ec62ddc9f6e39778 not found: ID does not exist" Oct 30 18:19:01.587458 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.587435 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istio-envoy\") pod \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " Oct 30 18:19:01.587555 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.587531 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-credential-socket\") pod \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " Oct 30 18:19:01.587615 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.587587 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-workload-socket\") pod \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " Oct 30 18:19:01.587682 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.587625 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-kube-api-access-zrllj\" (UniqueName: \"kubernetes.io/projected/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-kube-api-access-zrllj\") pod \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " Oct 30 18:19:01.587682 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.587655 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istiod-ca-cert\") pod \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " Oct 30 18:19:01.587792 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.587716 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-workload-certs\") pod \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " Oct 30 18:19:01.587792 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.587744 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istio-data\") pod \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " Oct 30 18:19:01.587792 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.587775 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istio-podinfo\") pod \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " Oct 30 18:19:01.587998 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.587829 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-credential-socket" pod "2a8f0904-a1a9-48fe-a499-5ec1f5d141c6" (UID: "2a8f0904-a1a9-48fe-a499-5ec1f5d141c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:01.587998 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.587845 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-workload-socket" pod "2a8f0904-a1a9-48fe-a499-5ec1f5d141c6" (UID: "2a8f0904-a1a9-48fe-a499-5ec1f5d141c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:01.587998 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.587833 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istio-token\" (UniqueName: \"kubernetes.io/projected/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istio-token\") pod \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\" (UID: \"2a8f0904-a1a9-48fe-a499-5ec1f5d141c6\") " Oct 30 18:19:01.588176 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.588045 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istio-data" pod "2a8f0904-a1a9-48fe-a499-5ec1f5d141c6" (UID: "2a8f0904-a1a9-48fe-a499-5ec1f5d141c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:01.588176 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.588051 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-workload-certs" pod "2a8f0904-a1a9-48fe-a499-5ec1f5d141c6" (UID: "2a8f0904-a1a9-48fe-a499-5ec1f5d141c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:01.588347 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.588323 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:19:01.588476 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.588339 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istiod-ca-cert" pod "2a8f0904-a1a9-48fe-a499-5ec1f5d141c6" (UID: "2a8f0904-a1a9-48fe-a499-5ec1f5d141c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:01.588476 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.588349 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:19:01.588476 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.588384 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:19:01.588476 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.588402 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:19:01.589966 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.589939 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istio-token" pod "2a8f0904-a1a9-48fe-a499-5ec1f5d141c6" (UID: "2a8f0904-a1a9-48fe-a499-5ec1f5d141c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:01.590048 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.589994 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istio-podinfo" pod "2a8f0904-a1a9-48fe-a499-5ec1f5d141c6" (UID: "2a8f0904-a1a9-48fe-a499-5ec1f5d141c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:19:01.590148 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.590118 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-kube-api-access-zrllj" pod "2a8f0904-a1a9-48fe-a499-5ec1f5d141c6" (UID: "2a8f0904-a1a9-48fe-a499-5ec1f5d141c6"). InnerVolumeSpecName "kube-api-access-zrllj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:01.590188 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.590165 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istio-envoy" pod "2a8f0904-a1a9-48fe-a499-5ec1f5d141c6" (UID: "2a8f0904-a1a9-48fe-a499-5ec1f5d141c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:01.689557 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.689523 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:19:01.689557 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.689552 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrllj\" (UniqueName: \"kubernetes.io/projected/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-kube-api-access-zrllj\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:19:01.689557 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.689562 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:19:01.689795 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.689572 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:19:01.689795 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.689580 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:19:01.898282 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.898248 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb"] Oct 30 18:19:01.910489 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:01.910459 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-05e969-openshift-default-b7f997b69-m9mrb"] Oct 30 18:19:02.100108 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:02.100064 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a8f0904-a1a9-48fe-a499-5ec1f5d141c6" path="/var/lib/kubelet/pods/2a8f0904-a1a9-48fe-a499-5ec1f5d141c6/volumes" Oct 30 18:19:05.995849 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:19:05.995768 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:19:08.137397 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:08.137360 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-05e969-c8cf69488k2sdl" podUID="e722b15b-5e82-47e5-b5f6-34d411f8f8fc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:19:09.752128 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:09.752093 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:19:12.549890 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:12.549846 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-05e969-c8cf69488k2sdl" Oct 30 18:19:12.626534 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:12.626501 4965 generic.go:356] "Generic (PLEG): container finished" podID="e722b15b-5e82-47e5-b5f6-34d411f8f8fc" containerID="dc1cc5e2cda5da3d3fdbe6e4d75fd13db8caf6e4b3750640d34ed3d4f288ecc6" exitCode=0 Oct 30 18:19:12.626701 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:12.626545 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-05e969-c8cf69488k2sdl" Oct 30 18:19:12.626701 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:12.626577 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-05e969-c8cf69488k2sdl" event={"ID":"e722b15b-5e82-47e5-b5f6-34d411f8f8fc","Type":"ContainerDied","Data":"dc1cc5e2cda5da3d3fdbe6e4d75fd13db8caf6e4b3750640d34ed3d4f288ecc6"} Oct 30 18:19:12.626701 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:12.626632 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-05e969-c8cf69488k2sdl" event={"ID":"e722b15b-5e82-47e5-b5f6-34d411f8f8fc","Type":"ContainerDied","Data":"45a6b83ce041cd154969ae9166c4a580e3747865ec5e0edc6ee23cd9bb7b3da2"} Oct 30 18:19:12.626701 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:12.626656 4965 scope.go:122] "RemoveContainer" containerID="dc1cc5e2cda5da3d3fdbe6e4d75fd13db8caf6e4b3750640d34ed3d4f288ecc6" Oct 30 18:19:12.638904 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:12.638855 4965 scope.go:122] "RemoveContainer" containerID="dc1cc5e2cda5da3d3fdbe6e4d75fd13db8caf6e4b3750640d34ed3d4f288ecc6" Oct 30 18:19:12.639205 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:19:12.639177 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc1cc5e2cda5da3d3fdbe6e4d75fd13db8caf6e4b3750640d34ed3d4f288ecc6\": container with ID starting with dc1cc5e2cda5da3d3fdbe6e4d75fd13db8caf6e4b3750640d34ed3d4f288ecc6 not found: ID does not exist" containerID="dc1cc5e2cda5da3d3fdbe6e4d75fd13db8caf6e4b3750640d34ed3d4f288ecc6" Oct 30 18:19:12.639295 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:12.639217 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc1cc5e2cda5da3d3fdbe6e4d75fd13db8caf6e4b3750640d34ed3d4f288ecc6"} err="failed to get container status \"dc1cc5e2cda5da3d3fdbe6e4d75fd13db8caf6e4b3750640d34ed3d4f288ecc6\": rpc error: code = NotFound desc = could not find container \"dc1cc5e2cda5da3d3fdbe6e4d75fd13db8caf6e4b3750640d34ed3d4f288ecc6\": container with ID starting with dc1cc5e2cda5da3d3fdbe6e4d75fd13db8caf6e4b3750640d34ed3d4f288ecc6 not found: ID does not exist" Oct 30 18:19:12.716825 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:12.716800 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e722b15b-5e82-47e5-b5f6-34d411f8f8fc-kube-api-access-557x6\" (UniqueName: \"kubernetes.io/projected/e722b15b-5e82-47e5-b5f6-34d411f8f8fc-kube-api-access-557x6\") pod \"e722b15b-5e82-47e5-b5f6-34d411f8f8fc\" (UID: \"e722b15b-5e82-47e5-b5f6-34d411f8f8fc\") " Oct 30 18:19:12.717005 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:12.716938 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e722b15b-5e82-47e5-b5f6-34d411f8f8fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e722b15b-5e82-47e5-b5f6-34d411f8f8fc-plugins-config-volume\") pod \"e722b15b-5e82-47e5-b5f6-34d411f8f8fc\" (UID: \"e722b15b-5e82-47e5-b5f6-34d411f8f8fc\") " Oct 30 18:19:12.717307 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:12.717287 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e722b15b-5e82-47e5-b5f6-34d411f8f8fc-plugins-config-volume" pod "e722b15b-5e82-47e5-b5f6-34d411f8f8fc" (UID: "e722b15b-5e82-47e5-b5f6-34d411f8f8fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:12.718852 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:12.718830 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e722b15b-5e82-47e5-b5f6-34d411f8f8fc-kube-api-access-557x6" pod "e722b15b-5e82-47e5-b5f6-34d411f8f8fc" (UID: "e722b15b-5e82-47e5-b5f6-34d411f8f8fc"). InnerVolumeSpecName "kube-api-access-557x6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:12.818142 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:12.818113 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e722b15b-5e82-47e5-b5f6-34d411f8f8fc-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:19:12.818142 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:12.818139 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-557x6\" (UniqueName: \"kubernetes.io/projected/e722b15b-5e82-47e5-b5f6-34d411f8f8fc-kube-api-access-557x6\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:19:12.970856 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:12.970822 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-05e969-c8cf69488k2sdl"] Oct 30 18:19:12.974340 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:12.974319 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-05e969-c8cf69488k2sdl"] Oct 30 18:19:14.097394 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:14.097350 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e722b15b-5e82-47e5-b5f6-34d411f8f8fc" path="/var/lib/kubelet/pods/e722b15b-5e82-47e5-b5f6-34d411f8f8fc/volumes" Oct 30 18:19:15.618725 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:19:15.618687 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:19:35.995712 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:19:35.995670 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:20:05.995619 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:20:05.995582 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:20:35.996138 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:20:35.996102 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:20:43.124851 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:20:43.124815 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:20:58.720685 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:20:58.720643 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:21:05.995312 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:21:05.995272 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:21:35.995481 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:21:35.995439 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:21:43.821690 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:43.821654 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td"] Oct 30 18:21:43.822184 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:43.822172 4965 cpu_manager.go:414] "containerMap: removing container" podUID="2a8f0904-a1a9-48fe-a499-5ec1f5d141c6" containerName="istio-proxy" Oct 30 18:21:43.822238 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:43.822185 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a8f0904-a1a9-48fe-a499-5ec1f5d141c6" containerName="istio-proxy" Oct 30 18:21:43.822238 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:43.822203 4965 cpu_manager.go:414] "containerMap: removing container" podUID="2a8f0904-a1a9-48fe-a499-5ec1f5d141c6" containerName="istio-proxy" Oct 30 18:21:43.822238 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:43.822211 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a8f0904-a1a9-48fe-a499-5ec1f5d141c6" containerName="istio-proxy" Oct 30 18:21:43.822238 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:43.822236 4965 cpu_manager.go:414] "containerMap: removing container" podUID="e722b15b-5e82-47e5-b5f6-34d411f8f8fc" containerName="payload-processing" Oct 30 18:21:43.822373 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:43.822241 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e722b15b-5e82-47e5-b5f6-34d411f8f8fc" containerName="payload-processing" Oct 30 18:21:43.822373 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:43.822302 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="e722b15b-5e82-47e5-b5f6-34d411f8f8fc" containerName="payload-processing" Oct 30 18:21:43.822373 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:43.822316 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a8f0904-a1a9-48fe-a499-5ec1f5d141c6" containerName="istio-proxy" Oct 30 18:21:43.822373 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:43.822323 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a8f0904-a1a9-48fe-a499-5ec1f5d141c6" containerName="istio-proxy" Oct 30 18:21:43.825632 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:43.825610 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:43.844620 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:43.844593 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-67382b-openshift-default-dockercfg-v74x5\"" Oct 30 18:21:43.856799 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:43.856774 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td"] Oct 30 18:21:43.929626 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:43.929593 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-credential-socket\") pod \"e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:43.929808 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:43.929657 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e26896af-c7b3-47bb-aca2-063e698acc37-istio-podinfo\") pod \"e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:43.929808 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:43.929750 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e26896af-c7b3-47bb-aca2-063e698acc37-istio-token\") pod \"e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:43.929808 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:43.929784 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-istio-data\") pod \"e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:43.929941 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:43.929813 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e26896af-c7b3-47bb-aca2-063e698acc37-istiod-ca-cert\") pod \"e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:43.929981 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:43.929935 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trrlj\" (UniqueName: \"kubernetes.io/projected/e26896af-c7b3-47bb-aca2-063e698acc37-kube-api-access-trrlj\") pod \"e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:43.930017 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:43.929986 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-istio-envoy\") pod \"e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:43.930057 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:43.930014 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-workload-socket\") pod \"e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:43.930093 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:43.930064 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-workload-certs\") pod \"e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:44.032309 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:44.032112 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e26896af-c7b3-47bb-aca2-063e698acc37-istio-podinfo\") pod \"e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:44.038340 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:44.033967 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e26896af-c7b3-47bb-aca2-063e698acc37-istio-token\") pod \"e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:44.038340 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:44.034028 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-istio-data\") pod \"e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:44.038340 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:44.034057 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e26896af-c7b3-47bb-aca2-063e698acc37-istiod-ca-cert\") pod \"e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:44.038340 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:44.034484 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trrlj\" (UniqueName: \"kubernetes.io/projected/e26896af-c7b3-47bb-aca2-063e698acc37-kube-api-access-trrlj\") pod \"e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:44.038340 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:44.034546 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-istio-envoy\") pod \"e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:44.038340 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:44.034589 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-workload-socket\") pod \"e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:44.038340 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:44.034615 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-workload-certs\") pod \"e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:44.038340 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:44.034667 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-credential-socket\") pod \"e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:44.039128 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:44.039073 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e26896af-c7b3-47bb-aca2-063e698acc37-istiod-ca-cert\") pod \"e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:44.039518 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:44.039386 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-workload-socket\") pod \"e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:44.040251 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:44.040220 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-workload-certs\") pod \"e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:44.040447 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:44.040424 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-credential-socket\") pod \"e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:44.040585 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:44.040562 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-istio-data\") pod \"e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:44.041588 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:44.041562 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e26896af-c7b3-47bb-aca2-063e698acc37-istio-podinfo\") pod \"e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:44.042699 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:44.042674 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-istio-envoy\") pod \"e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:44.057876 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:44.057823 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e26896af-c7b3-47bb-aca2-063e698acc37-istio-token\") pod \"e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:44.058010 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:44.057973 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trrlj\" (UniqueName: \"kubernetes.io/projected/e26896af-c7b3-47bb-aca2-063e698acc37-kube-api-access-trrlj\") pod \"e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:44.139617 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:44.139579 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:44.463913 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:44.463888 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td"] Oct 30 18:21:44.465665 ip-10-0-139-222 kubenswrapper[4965]: W1030 18:21:44.465635 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode26896af_c7b3_47bb_aca2_063e698acc37.slice/crio-25d1890284373e267045811f7273f0d670ab992d751d359f48bf4f2b2e52b44d WatchSource:0}: Error finding container 25d1890284373e267045811f7273f0d670ab992d751d359f48bf4f2b2e52b44d: Status 404 returned error can't find the container with id 25d1890284373e267045811f7273f0d670ab992d751d359f48bf4f2b2e52b44d Oct 30 18:21:44.467753 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:44.467720 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 18:21:44.467807 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:44.467786 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 18:21:44.467850 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:44.467818 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 18:21:45.403185 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:45.403146 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" event={"ID":"e26896af-c7b3-47bb-aca2-063e698acc37","Type":"ContainerStarted","Data":"bbbd307454b86f058c0b55c5d5f871db8f4b2a3846a1e01ca19d5ee51cc2dc04"} Oct 30 18:21:45.403698 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:45.403216 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" event={"ID":"e26896af-c7b3-47bb-aca2-063e698acc37","Type":"ContainerStarted","Data":"25d1890284373e267045811f7273f0d670ab992d751d359f48bf4f2b2e52b44d"} Oct 30 18:21:45.452635 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:45.452573 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" podStartSLOduration=2.452558198 podStartE2EDuration="2.452558198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:21:45.450412321 +0000 UTC m=+4163.502512263" watchObservedRunningTime="2026-10-30 18:21:45.452558198 +0000 UTC m=+4163.504658131" Oct 30 18:21:46.140485 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:46.140452 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:46.145428 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:46.145400 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:46.408396 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:46.408318 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:46.409265 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:46.409243 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:21:49.689567 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:49.689533 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg"] Oct 30 18:21:49.693526 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:49.693504 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg" Oct 30 18:21:49.720625 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:49.720592 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-67382b\"" Oct 30 18:21:49.722636 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:49.722610 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg"] Oct 30 18:21:49.810909 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:49.810837 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvkgm\" (UniqueName: \"kubernetes.io/projected/30a1c12f-e574-4ea3-9ab7-3eb9319882b0-kube-api-access-bvkgm\") pod \"payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg\" (UID: \"30a1c12f-e574-4ea3-9ab7-3eb9319882b0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg" Oct 30 18:21:49.811398 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:49.811370 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30a1c12f-e574-4ea3-9ab7-3eb9319882b0-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg\" (UID: \"30a1c12f-e574-4ea3-9ab7-3eb9319882b0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg" Oct 30 18:21:49.913034 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:49.912996 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30a1c12f-e574-4ea3-9ab7-3eb9319882b0-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg\" (UID: \"30a1c12f-e574-4ea3-9ab7-3eb9319882b0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg" Oct 30 18:21:49.913232 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:49.913059 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvkgm\" (UniqueName: \"kubernetes.io/projected/30a1c12f-e574-4ea3-9ab7-3eb9319882b0-kube-api-access-bvkgm\") pod \"payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg\" (UID: \"30a1c12f-e574-4ea3-9ab7-3eb9319882b0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg" Oct 30 18:21:49.913738 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:49.913713 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30a1c12f-e574-4ea3-9ab7-3eb9319882b0-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg\" (UID: \"30a1c12f-e574-4ea3-9ab7-3eb9319882b0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg" Oct 30 18:21:49.936481 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:49.936441 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvkgm\" (UniqueName: \"kubernetes.io/projected/30a1c12f-e574-4ea3-9ab7-3eb9319882b0-kube-api-access-bvkgm\") pod \"payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg\" (UID: \"30a1c12f-e574-4ea3-9ab7-3eb9319882b0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg" Oct 30 18:21:50.007477 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:50.007373 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg" Oct 30 18:21:50.311976 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:50.311948 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg"] Oct 30 18:21:50.313793 ip-10-0-139-222 kubenswrapper[4965]: W1030 18:21:50.313766 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a1c12f_e574_4ea3_9ab7_3eb9319882b0.slice/crio-325a4f227bedc838ccb4a93230baeb7ca1595d8db239232a77d2c4d4e6075c05 WatchSource:0}: Error finding container 325a4f227bedc838ccb4a93230baeb7ca1595d8db239232a77d2c4d4e6075c05: Status 404 returned error can't find the container with id 325a4f227bedc838ccb4a93230baeb7ca1595d8db239232a77d2c4d4e6075c05 Oct 30 18:21:50.433535 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:50.433500 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg" event={"ID":"30a1c12f-e574-4ea3-9ab7-3eb9319882b0","Type":"ContainerStarted","Data":"3ce88efffea6c8e1b7e20076bf8fc5b0c6d7ce947ef2c501dd8009571035f85c"} Oct 30 18:21:50.433751 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:50.433551 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg" event={"ID":"30a1c12f-e574-4ea3-9ab7-3eb9319882b0","Type":"ContainerStarted","Data":"325a4f227bedc838ccb4a93230baeb7ca1595d8db239232a77d2c4d4e6075c05"} Oct 30 18:21:50.433751 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:50.433631 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg" Oct 30 18:21:50.473535 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:50.473481 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg" podStartSLOduration=1.47346825 podStartE2EDuration="1.47346825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:21:50.473053644 +0000 UTC m=+4168.525153575" watchObservedRunningTime="2026-10-30 18:21:50.47346825 +0000 UTC m=+4168.525568181" Oct 30 18:21:52.537598 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:21:52.537559 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:22:01.440908 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:22:01.440855 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg" Oct 30 18:22:02.280848 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:22:02.279588 4965 patch_prober.go:29] interesting pod/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": read tcp 10.132.0.2:49328->10.132.0.230:15021: read: connection reset by peer" start-of-body= Oct 30 18:22:02.280848 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:22:02.279660 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" podUID="e26896af-c7b3-47bb-aca2-063e698acc37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": read tcp 10.132.0.2:49328->10.132.0.230:15021: read: connection reset by peer" Oct 30 18:22:02.529453 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:22:02.529350 4965 generic.go:356] "Generic (PLEG): container finished" podID="e26896af-c7b3-47bb-aca2-063e698acc37" containerID="bbbd307454b86f058c0b55c5d5f871db8f4b2a3846a1e01ca19d5ee51cc2dc04" exitCode=0 Oct 30 18:22:02.529453 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:22:02.529426 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" event={"ID":"e26896af-c7b3-47bb-aca2-063e698acc37","Type":"ContainerDied","Data":"bbbd307454b86f058c0b55c5d5f871db8f4b2a3846a1e01ca19d5ee51cc2dc04"} Oct 30 18:22:02.529892 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:22:02.529845 4965 scope.go:122] "RemoveContainer" containerID="bbbd307454b86f058c0b55c5d5f871db8f4b2a3846a1e01ca19d5ee51cc2dc04" Oct 30 18:22:02.531043 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:22:02.531004 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 18:22:02.531161 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:22:02.531074 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 18:22:02.531161 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:22:02.531102 4965 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 18:22:03.536416 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:22:03.536335 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" event={"ID":"e26896af-c7b3-47bb-aca2-063e698acc37","Type":"ContainerStarted","Data":"3d1219ce145738be6cf4c8ab4c1b098d3b6c75bc4b917521a0936a64418bf6ab"} Oct 30 18:22:04.139712 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:22:04.139681 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:22:04.139961 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:22:04.139944 4965 patch_prober.go:29] interesting pod/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 18:22:04.140028 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:22:04.139990 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" podUID="e26896af-c7b3-47bb-aca2-063e698acc37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 18:22:05.140993 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:22:05.140956 4965 patch_prober.go:29] interesting pod/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 18:22:05.141396 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:22:05.141017 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" podUID="e26896af-c7b3-47bb-aca2-063e698acc37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 18:22:06.006654 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:22:06.006622 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:22:06.140706 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:22:06.140671 4965 patch_prober.go:29] interesting pod/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 18:22:06.140918 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:22:06.140721 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" podUID="e26896af-c7b3-47bb-aca2-063e698acc37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 18:22:07.139943 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:22:07.139910 4965 patch_prober.go:29] interesting pod/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 18:22:07.140332 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:22:07.139960 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" podUID="e26896af-c7b3-47bb-aca2-063e698acc37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 18:22:08.031308 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:22:08.031275 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:22:08.140545 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:22:08.140509 4965 patch_prober.go:29] interesting pod/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 18:22:08.140981 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:22:08.140568 4965 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" podUID="e26896af-c7b3-47bb-aca2-063e698acc37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 18:22:09.144037 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:22:09.144001 4965 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:22:09.144513 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:22:09.144289 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:22:09.145083 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:22:09.145067 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:22:22.984251 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:22:22.984221 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 18:22:22.985151 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:22:22.985129 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 18:22:32.614278 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:22:32.614240 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:22:35.996029 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:22:35.995994 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:22:46.258185 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:22:46.258150 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:22:56.339473 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:22:56.339439 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg"] Oct 30 18:23:00.831997 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:00.831959 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:23:01.339801 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:01.339763 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg" podUID="30a1c12f-e574-4ea3-9ab7-3eb9319882b0" containerName="payload-processing" containerID="cri-o://3ce88efffea6c8e1b7e20076bf8fc5b0c6d7ce947ef2c501dd8009571035f85c" gracePeriod=25 Oct 30 18:23:01.439917 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:01.439846 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg" podUID="30a1c12f-e574-4ea3-9ab7-3eb9319882b0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:23:05.997042 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:23:05.997002 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:23:06.102195 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:06.102154 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td"] Oct 30 18:23:06.102507 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:06.102448 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" podUID="e26896af-c7b3-47bb-aca2-063e698acc37" containerName="istio-proxy" containerID="cri-o://3d1219ce145738be6cf4c8ab4c1b098d3b6c75bc4b917521a0936a64418bf6ab" gracePeriod=30 Oct 30 18:23:09.147710 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:09.147666 4965 patch_prober.go:29] interesting pod/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:23:09.148213 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:09.147752 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" podUID="e26896af-c7b3-47bb-aca2-063e698acc37" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:23:11.367790 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.367726 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:23:11.439408 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.439366 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg" podUID="30a1c12f-e574-4ea3-9ab7-3eb9319882b0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:23:11.468919 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.468886 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-workload-certs\") pod \"e26896af-c7b3-47bb-aca2-063e698acc37\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " Oct 30 18:23:11.469090 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.468936 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e26896af-c7b3-47bb-aca2-063e698acc37-istio-token\" (UniqueName: \"kubernetes.io/projected/e26896af-c7b3-47bb-aca2-063e698acc37-istio-token\") pod \"e26896af-c7b3-47bb-aca2-063e698acc37\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " Oct 30 18:23:11.469090 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.468969 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-istio-data\") pod \"e26896af-c7b3-47bb-aca2-063e698acc37\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " Oct 30 18:23:11.469090 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.469005 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e26896af-c7b3-47bb-aca2-063e698acc37-kube-api-access-trrlj\" (UniqueName: \"kubernetes.io/projected/e26896af-c7b3-47bb-aca2-063e698acc37-kube-api-access-trrlj\") pod \"e26896af-c7b3-47bb-aca2-063e698acc37\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " Oct 30 18:23:11.469090 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.469064 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-credential-socket\") pod \"e26896af-c7b3-47bb-aca2-063e698acc37\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " Oct 30 18:23:11.469319 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.469084 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-workload-certs" pod "e26896af-c7b3-47bb-aca2-063e698acc37" (UID: "e26896af-c7b3-47bb-aca2-063e698acc37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:11.469319 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.469093 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-workload-socket\") pod \"e26896af-c7b3-47bb-aca2-063e698acc37\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " Oct 30 18:23:11.469319 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.469174 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e26896af-c7b3-47bb-aca2-063e698acc37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e26896af-c7b3-47bb-aca2-063e698acc37-istiod-ca-cert\") pod \"e26896af-c7b3-47bb-aca2-063e698acc37\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " Oct 30 18:23:11.469319 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.469235 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-istio-envoy\") pod \"e26896af-c7b3-47bb-aca2-063e698acc37\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " Oct 30 18:23:11.469548 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.469337 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e26896af-c7b3-47bb-aca2-063e698acc37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e26896af-c7b3-47bb-aca2-063e698acc37-istio-podinfo\") pod \"e26896af-c7b3-47bb-aca2-063e698acc37\" (UID: \"e26896af-c7b3-47bb-aca2-063e698acc37\") " Oct 30 18:23:11.469548 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.469336 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-istio-data" pod "e26896af-c7b3-47bb-aca2-063e698acc37" (UID: "e26896af-c7b3-47bb-aca2-063e698acc37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:11.469548 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.469366 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-workload-socket" pod "e26896af-c7b3-47bb-aca2-063e698acc37" (UID: "e26896af-c7b3-47bb-aca2-063e698acc37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:11.469548 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.469428 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-credential-socket" pod "e26896af-c7b3-47bb-aca2-063e698acc37" (UID: "e26896af-c7b3-47bb-aca2-063e698acc37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:11.469849 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.469618 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e26896af-c7b3-47bb-aca2-063e698acc37-istiod-ca-cert" pod "e26896af-c7b3-47bb-aca2-063e698acc37" (UID: "e26896af-c7b3-47bb-aca2-063e698acc37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:23:11.469849 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.469807 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e26896af-c7b3-47bb-aca2-063e698acc37-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:23:11.469849 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.469830 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:23:11.469849 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.469844 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:23:11.470096 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.469857 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:23:11.470096 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.469886 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:23:11.471498 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.471469 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e26896af-c7b3-47bb-aca2-063e698acc37-istio-podinfo" pod "e26896af-c7b3-47bb-aca2-063e698acc37" (UID: "e26896af-c7b3-47bb-aca2-063e698acc37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:23:11.471708 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.471687 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-istio-envoy" pod "e26896af-c7b3-47bb-aca2-063e698acc37" (UID: "e26896af-c7b3-47bb-aca2-063e698acc37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:11.471777 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.471714 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e26896af-c7b3-47bb-aca2-063e698acc37-kube-api-access-trrlj" pod "e26896af-c7b3-47bb-aca2-063e698acc37" (UID: "e26896af-c7b3-47bb-aca2-063e698acc37"). InnerVolumeSpecName "kube-api-access-trrlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:11.472029 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.472009 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e26896af-c7b3-47bb-aca2-063e698acc37-istio-token" pod "e26896af-c7b3-47bb-aca2-063e698acc37" (UID: "e26896af-c7b3-47bb-aca2-063e698acc37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:11.570805 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.570769 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e26896af-c7b3-47bb-aca2-063e698acc37-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:23:11.570805 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.570798 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trrlj\" (UniqueName: \"kubernetes.io/projected/e26896af-c7b3-47bb-aca2-063e698acc37-kube-api-access-trrlj\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:23:11.570805 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.570809 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e26896af-c7b3-47bb-aca2-063e698acc37-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:23:11.571061 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.570817 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e26896af-c7b3-47bb-aca2-063e698acc37-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:23:11.892157 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.892125 4965 generic.go:356] "Generic (PLEG): container finished" podID="e26896af-c7b3-47bb-aca2-063e698acc37" containerID="3d1219ce145738be6cf4c8ab4c1b098d3b6c75bc4b917521a0936a64418bf6ab" exitCode=0 Oct 30 18:23:11.892313 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.892173 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" Oct 30 18:23:11.892313 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.892179 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" event={"ID":"e26896af-c7b3-47bb-aca2-063e698acc37","Type":"ContainerDied","Data":"3d1219ce145738be6cf4c8ab4c1b098d3b6c75bc4b917521a0936a64418bf6ab"} Oct 30 18:23:11.892313 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.892300 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td" event={"ID":"e26896af-c7b3-47bb-aca2-063e698acc37","Type":"ContainerDied","Data":"25d1890284373e267045811f7273f0d670ab992d751d359f48bf4f2b2e52b44d"} Oct 30 18:23:11.892445 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.892329 4965 scope.go:122] "RemoveContainer" containerID="3d1219ce145738be6cf4c8ab4c1b098d3b6c75bc4b917521a0936a64418bf6ab" Oct 30 18:23:11.902397 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.902382 4965 scope.go:122] "RemoveContainer" containerID="bbbd307454b86f058c0b55c5d5f871db8f4b2a3846a1e01ca19d5ee51cc2dc04" Oct 30 18:23:11.910351 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.910334 4965 scope.go:122] "RemoveContainer" containerID="3d1219ce145738be6cf4c8ab4c1b098d3b6c75bc4b917521a0936a64418bf6ab" Oct 30 18:23:11.910609 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:23:11.910588 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d1219ce145738be6cf4c8ab4c1b098d3b6c75bc4b917521a0936a64418bf6ab\": container with ID starting with 3d1219ce145738be6cf4c8ab4c1b098d3b6c75bc4b917521a0936a64418bf6ab not found: ID does not exist" containerID="3d1219ce145738be6cf4c8ab4c1b098d3b6c75bc4b917521a0936a64418bf6ab" Oct 30 18:23:11.910665 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.910615 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d1219ce145738be6cf4c8ab4c1b098d3b6c75bc4b917521a0936a64418bf6ab"} err="failed to get container status \"3d1219ce145738be6cf4c8ab4c1b098d3b6c75bc4b917521a0936a64418bf6ab\": rpc error: code = NotFound desc = could not find container \"3d1219ce145738be6cf4c8ab4c1b098d3b6c75bc4b917521a0936a64418bf6ab\": container with ID starting with 3d1219ce145738be6cf4c8ab4c1b098d3b6c75bc4b917521a0936a64418bf6ab not found: ID does not exist" Oct 30 18:23:11.910665 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.910637 4965 scope.go:122] "RemoveContainer" containerID="bbbd307454b86f058c0b55c5d5f871db8f4b2a3846a1e01ca19d5ee51cc2dc04" Oct 30 18:23:11.910901 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:23:11.910881 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbbd307454b86f058c0b55c5d5f871db8f4b2a3846a1e01ca19d5ee51cc2dc04\": container with ID starting with bbbd307454b86f058c0b55c5d5f871db8f4b2a3846a1e01ca19d5ee51cc2dc04 not found: ID does not exist" containerID="bbbd307454b86f058c0b55c5d5f871db8f4b2a3846a1e01ca19d5ee51cc2dc04" Oct 30 18:23:11.910953 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.910908 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbbd307454b86f058c0b55c5d5f871db8f4b2a3846a1e01ca19d5ee51cc2dc04"} err="failed to get container status \"bbbd307454b86f058c0b55c5d5f871db8f4b2a3846a1e01ca19d5ee51cc2dc04\": rpc error: code = NotFound desc = could not find container \"bbbd307454b86f058c0b55c5d5f871db8f4b2a3846a1e01ca19d5ee51cc2dc04\": container with ID starting with bbbd307454b86f058c0b55c5d5f871db8f4b2a3846a1e01ca19d5ee51cc2dc04 not found: ID does not exist" Oct 30 18:23:11.955913 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.955884 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td"] Oct 30 18:23:11.968432 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:11.968407 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-67382b-openshift-default-684b4b87bffz9td"] Oct 30 18:23:12.097215 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:12.097173 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e26896af-c7b3-47bb-aca2-063e698acc37" path="/var/lib/kubelet/pods/e26896af-c7b3-47bb-aca2-063e698acc37/volumes" Oct 30 18:23:21.445833 ip-10-0-139-222 kubenswrapper[4965]: W1030 18:23:21.445804 4965 logging.go:55] [core] [Channel #12558 SubChannel #12559]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.231:50052", ServerName: "10.132.0.231:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.231:50052: connect: connection refused" Oct 30 18:23:21.601480 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:21.601421 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg" Oct 30 18:23:21.771389 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:21.771347 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30a1c12f-e574-4ea3-9ab7-3eb9319882b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30a1c12f-e574-4ea3-9ab7-3eb9319882b0-plugins-config-volume\") pod \"30a1c12f-e574-4ea3-9ab7-3eb9319882b0\" (UID: \"30a1c12f-e574-4ea3-9ab7-3eb9319882b0\") " Oct 30 18:23:21.771571 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:21.771467 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30a1c12f-e574-4ea3-9ab7-3eb9319882b0-kube-api-access-bvkgm\" (UniqueName: \"kubernetes.io/projected/30a1c12f-e574-4ea3-9ab7-3eb9319882b0-kube-api-access-bvkgm\") pod \"30a1c12f-e574-4ea3-9ab7-3eb9319882b0\" (UID: \"30a1c12f-e574-4ea3-9ab7-3eb9319882b0\") " Oct 30 18:23:21.771740 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:21.771717 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30a1c12f-e574-4ea3-9ab7-3eb9319882b0-plugins-config-volume" pod "30a1c12f-e574-4ea3-9ab7-3eb9319882b0" (UID: "30a1c12f-e574-4ea3-9ab7-3eb9319882b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:23:21.773601 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:21.773575 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30a1c12f-e574-4ea3-9ab7-3eb9319882b0-kube-api-access-bvkgm" pod "30a1c12f-e574-4ea3-9ab7-3eb9319882b0" (UID: "30a1c12f-e574-4ea3-9ab7-3eb9319882b0"). InnerVolumeSpecName "kube-api-access-bvkgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:21.872688 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:21.872615 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvkgm\" (UniqueName: \"kubernetes.io/projected/30a1c12f-e574-4ea3-9ab7-3eb9319882b0-kube-api-access-bvkgm\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:23:21.872688 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:21.872642 4965 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30a1c12f-e574-4ea3-9ab7-3eb9319882b0-plugins-config-volume\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:23:21.940285 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:21.940249 4965 generic.go:356] "Generic (PLEG): container finished" podID="30a1c12f-e574-4ea3-9ab7-3eb9319882b0" containerID="3ce88efffea6c8e1b7e20076bf8fc5b0c6d7ce947ef2c501dd8009571035f85c" exitCode=0 Oct 30 18:23:21.940480 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:21.940306 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg" Oct 30 18:23:21.940480 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:21.940335 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg" event={"ID":"30a1c12f-e574-4ea3-9ab7-3eb9319882b0","Type":"ContainerDied","Data":"3ce88efffea6c8e1b7e20076bf8fc5b0c6d7ce947ef2c501dd8009571035f85c"} Oct 30 18:23:21.940480 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:21.940386 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg" event={"ID":"30a1c12f-e574-4ea3-9ab7-3eb9319882b0","Type":"ContainerDied","Data":"325a4f227bedc838ccb4a93230baeb7ca1595d8db239232a77d2c4d4e6075c05"} Oct 30 18:23:21.940480 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:21.940405 4965 scope.go:122] "RemoveContainer" containerID="3ce88efffea6c8e1b7e20076bf8fc5b0c6d7ce947ef2c501dd8009571035f85c" Oct 30 18:23:21.949932 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:21.949916 4965 scope.go:122] "RemoveContainer" containerID="3ce88efffea6c8e1b7e20076bf8fc5b0c6d7ce947ef2c501dd8009571035f85c" Oct 30 18:23:21.950193 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:23:21.950170 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ce88efffea6c8e1b7e20076bf8fc5b0c6d7ce947ef2c501dd8009571035f85c\": container with ID starting with 3ce88efffea6c8e1b7e20076bf8fc5b0c6d7ce947ef2c501dd8009571035f85c not found: ID does not exist" containerID="3ce88efffea6c8e1b7e20076bf8fc5b0c6d7ce947ef2c501dd8009571035f85c" Oct 30 18:23:21.950264 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:21.950203 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ce88efffea6c8e1b7e20076bf8fc5b0c6d7ce947ef2c501dd8009571035f85c"} err="failed to get container status \"3ce88efffea6c8e1b7e20076bf8fc5b0c6d7ce947ef2c501dd8009571035f85c\": rpc error: code = NotFound desc = could not find container \"3ce88efffea6c8e1b7e20076bf8fc5b0c6d7ce947ef2c501dd8009571035f85c\": container with ID starting with 3ce88efffea6c8e1b7e20076bf8fc5b0c6d7ce947ef2c501dd8009571035f85c not found: ID does not exist" Oct 30 18:23:21.989509 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:21.989480 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg"] Oct 30 18:23:22.001623 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:22.001600 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg"] Oct 30 18:23:22.098471 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:22.098441 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30a1c12f-e574-4ea3-9ab7-3eb9319882b0" path="/var/lib/kubelet/pods/30a1c12f-e574-4ea3-9ab7-3eb9319882b0/volumes" Oct 30 18:23:22.440002 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:22.439963 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-67382b-686f6f9d9whwpg" podUID="30a1c12f-e574-4ea3-9ab7-3eb9319882b0" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.231:50052\" within 1s: context deadline exceeded" Oct 30 18:23:23.340826 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:23.340791 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:23:35.995970 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:23:35.995929 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:23:44.611630 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:44.611593 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:23:47.451015 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:47.450979 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-pvlkq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:23:47.451398 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:23:47.451037 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" podUID="0c1403e9-4637-49cc-9e50-f7be9caa1847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:05.994993 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:24:05.994952 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:24:11.039954 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:24:11.039918 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:24:22.936956 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:24:22.936924 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:24:30.170909 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:24:30.167430 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rkm25 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:24:30.170909 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:24:30.167503 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" podUID="9e0d78c2-d701-4ecc-888c-d69469f844ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:31.031777 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:24:31.031738 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:24:35.995225 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:24:35.995190 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:24:38.920562 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:24:38.920526 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:24:52.407938 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:24:52.407904 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:24:54.710886 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:24:54.709967 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ms6c5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:24:54.710886 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:24:54.710073 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" podUID="f835cdef-6be0-4f7d-b4c1-6e5a771e8616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:05.995736 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:25:05.995650 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:25:29.542402 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:29.542363 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:25:30.683682 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:30.683650 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-9d288"] Oct 30 18:25:30.684246 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:30.684145 4965 cpu_manager.go:414] "containerMap: removing container" podUID="30a1c12f-e574-4ea3-9ab7-3eb9319882b0" containerName="payload-processing" Oct 30 18:25:30.684246 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:30.684159 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30a1c12f-e574-4ea3-9ab7-3eb9319882b0" containerName="payload-processing" Oct 30 18:25:30.684246 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:30.684172 4965 cpu_manager.go:414] "containerMap: removing container" podUID="e26896af-c7b3-47bb-aca2-063e698acc37" containerName="istio-proxy" Oct 30 18:25:30.684246 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:30.684177 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e26896af-c7b3-47bb-aca2-063e698acc37" containerName="istio-proxy" Oct 30 18:25:30.684246 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:30.684186 4965 cpu_manager.go:414] "containerMap: removing container" podUID="e26896af-c7b3-47bb-aca2-063e698acc37" containerName="istio-proxy" Oct 30 18:25:30.684246 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:30.684192 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e26896af-c7b3-47bb-aca2-063e698acc37" containerName="istio-proxy" Oct 30 18:25:30.685045 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:30.685016 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="e26896af-c7b3-47bb-aca2-063e698acc37" containerName="istio-proxy" Oct 30 18:25:30.685178 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:30.685050 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="e26896af-c7b3-47bb-aca2-063e698acc37" containerName="istio-proxy" Oct 30 18:25:30.685350 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:30.685332 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="30a1c12f-e574-4ea3-9ab7-3eb9319882b0" containerName="payload-processing" Oct 30 18:25:30.692886 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:30.692839 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-9d288" Oct 30 18:25:30.713294 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:30.712075 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-9d288"] Oct 30 18:25:30.744142 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:30.744105 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c0219a-9aa4-434b-ab8d-b6385fbf58bc-openshift-service-ca\") pod \"maas-api-595cc648c-9d288\" (UID: \"a4c0219a-9aa4-434b-ab8d-b6385fbf58bc\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-9d288" Oct 30 18:25:30.744413 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:30.744259 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4c0219a-9aa4-434b-ab8d-b6385fbf58bc-metrics-certs\") pod \"maas-api-595cc648c-9d288\" (UID: \"a4c0219a-9aa4-434b-ab8d-b6385fbf58bc\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-9d288" Oct 30 18:25:30.744413 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:30.744311 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4d2b\" (UniqueName: \"kubernetes.io/projected/a4c0219a-9aa4-434b-ab8d-b6385fbf58bc-kube-api-access-q4d2b\") pod \"maas-api-595cc648c-9d288\" (UID: \"a4c0219a-9aa4-434b-ab8d-b6385fbf58bc\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-9d288" Oct 30 18:25:30.744413 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:30.744330 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c0219a-9aa4-434b-ab8d-b6385fbf58bc-maas-api-tls\") pod \"maas-api-595cc648c-9d288\" (UID: \"a4c0219a-9aa4-434b-ab8d-b6385fbf58bc\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-9d288" Oct 30 18:25:30.845616 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:30.845577 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c0219a-9aa4-434b-ab8d-b6385fbf58bc-openshift-service-ca\") pod \"maas-api-595cc648c-9d288\" (UID: \"a4c0219a-9aa4-434b-ab8d-b6385fbf58bc\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-9d288" Oct 30 18:25:30.845831 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:30.845692 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4c0219a-9aa4-434b-ab8d-b6385fbf58bc-metrics-certs\") pod \"maas-api-595cc648c-9d288\" (UID: \"a4c0219a-9aa4-434b-ab8d-b6385fbf58bc\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-9d288" Oct 30 18:25:30.845831 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:30.845749 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4d2b\" (UniqueName: \"kubernetes.io/projected/a4c0219a-9aa4-434b-ab8d-b6385fbf58bc-kube-api-access-q4d2b\") pod \"maas-api-595cc648c-9d288\" (UID: \"a4c0219a-9aa4-434b-ab8d-b6385fbf58bc\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-9d288" Oct 30 18:25:30.845831 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:30.845773 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c0219a-9aa4-434b-ab8d-b6385fbf58bc-maas-api-tls\") pod \"maas-api-595cc648c-9d288\" (UID: \"a4c0219a-9aa4-434b-ab8d-b6385fbf58bc\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-9d288" Oct 30 18:25:30.846286 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:30.846265 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c0219a-9aa4-434b-ab8d-b6385fbf58bc-openshift-service-ca\") pod \"maas-api-595cc648c-9d288\" (UID: \"a4c0219a-9aa4-434b-ab8d-b6385fbf58bc\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-9d288" Oct 30 18:25:30.848141 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:30.848115 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4c0219a-9aa4-434b-ab8d-b6385fbf58bc-metrics-certs\") pod \"maas-api-595cc648c-9d288\" (UID: \"a4c0219a-9aa4-434b-ab8d-b6385fbf58bc\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-9d288" Oct 30 18:25:30.848272 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:30.848251 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c0219a-9aa4-434b-ab8d-b6385fbf58bc-maas-api-tls\") pod \"maas-api-595cc648c-9d288\" (UID: \"a4c0219a-9aa4-434b-ab8d-b6385fbf58bc\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-9d288" Oct 30 18:25:30.868400 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:30.868372 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4d2b\" (UniqueName: \"kubernetes.io/projected/a4c0219a-9aa4-434b-ab8d-b6385fbf58bc-kube-api-access-q4d2b\") pod \"maas-api-595cc648c-9d288\" (UID: \"a4c0219a-9aa4-434b-ab8d-b6385fbf58bc\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-9d288" Oct 30 18:25:31.018676 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:31.018598 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-9d288" Oct 30 18:25:31.320453 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:31.320430 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-9d288"] Oct 30 18:25:31.322382 ip-10-0-139-222 kubenswrapper[4965]: W1030 18:25:31.322351 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4c0219a_9aa4_434b_ab8d_b6385fbf58bc.slice/crio-fd384d71b5a99f82acfe867b2e68ae8ef4323337592fe4d607ac6d1f3d0b370d WatchSource:0}: Error finding container fd384d71b5a99f82acfe867b2e68ae8ef4323337592fe4d607ac6d1f3d0b370d: Status 404 returned error can't find the container with id fd384d71b5a99f82acfe867b2e68ae8ef4323337592fe4d607ac6d1f3d0b370d Oct 30 18:25:31.323747 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:31.323728 4965 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:25:31.698749 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:31.698713 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-9d288" event={"ID":"a4c0219a-9aa4-434b-ab8d-b6385fbf58bc","Type":"ContainerStarted","Data":"fd384d71b5a99f82acfe867b2e68ae8ef4323337592fe4d607ac6d1f3d0b370d"} Oct 30 18:25:32.705086 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:32.705048 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-9d288" event={"ID":"a4c0219a-9aa4-434b-ab8d-b6385fbf58bc","Type":"ContainerStarted","Data":"7cc86ad92f9b8d6a2c9612949dae716b7563476a33a4199f3e6646ff595e2492"} Oct 30 18:25:32.705481 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:32.705181 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-9d288" Oct 30 18:25:32.750569 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:32.750518 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-9d288" podStartSLOduration=2.33907251 podStartE2EDuration="2.750505605s" totalImagesPullingTime="411.433095ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:32.749887864 +0000 UTC m=+4390.801987796" watchObservedRunningTime="2026-10-30 18:25:32.750505605 +0000 UTC m=+4390.802605532" Oct 30 18:25:35.995409 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:25:35.995364 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:25:38.715689 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:38.715660 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-9d288" Oct 30 18:25:38.796879 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:38.795706 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-6tb6z"] Oct 30 18:25:38.796879 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:38.795981 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-6tb6z" podUID="c2f9d33f-085d-4744-85cc-d309f9f0ccad" containerName="maas-api" containerID="cri-o://12f071f9e0f895805ac2ee7dfc3225b7d06c08231a38be734410754a6c2eb9c5" gracePeriod=30 Oct 30 18:25:39.523063 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:39.523027 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:25:40.065185 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:40.065159 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-6tb6z" Oct 30 18:25:40.239741 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:40.239652 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2f9d33f-085d-4744-85cc-d309f9f0ccad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2f9d33f-085d-4744-85cc-d309f9f0ccad-maas-api-tls\") pod \"c2f9d33f-085d-4744-85cc-d309f9f0ccad\" (UID: \"c2f9d33f-085d-4744-85cc-d309f9f0ccad\") " Oct 30 18:25:40.239939 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:40.239788 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2f9d33f-085d-4744-85cc-d309f9f0ccad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2f9d33f-085d-4744-85cc-d309f9f0ccad-openshift-service-ca\") pod \"c2f9d33f-085d-4744-85cc-d309f9f0ccad\" (UID: \"c2f9d33f-085d-4744-85cc-d309f9f0ccad\") " Oct 30 18:25:40.239939 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:40.239833 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2f9d33f-085d-4744-85cc-d309f9f0ccad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2f9d33f-085d-4744-85cc-d309f9f0ccad-metrics-certs\") pod \"c2f9d33f-085d-4744-85cc-d309f9f0ccad\" (UID: \"c2f9d33f-085d-4744-85cc-d309f9f0ccad\") " Oct 30 18:25:40.240073 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:40.239999 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2f9d33f-085d-4744-85cc-d309f9f0ccad-kube-api-access-wm85w\" (UniqueName: \"kubernetes.io/projected/c2f9d33f-085d-4744-85cc-d309f9f0ccad-kube-api-access-wm85w\") pod \"c2f9d33f-085d-4744-85cc-d309f9f0ccad\" (UID: \"c2f9d33f-085d-4744-85cc-d309f9f0ccad\") " Oct 30 18:25:40.240743 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:40.240251 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2f9d33f-085d-4744-85cc-d309f9f0ccad-openshift-service-ca" pod "c2f9d33f-085d-4744-85cc-d309f9f0ccad" (UID: "c2f9d33f-085d-4744-85cc-d309f9f0ccad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:40.240743 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:40.240496 4965 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2f9d33f-085d-4744-85cc-d309f9f0ccad-openshift-service-ca\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:25:40.242298 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:40.242272 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2f9d33f-085d-4744-85cc-d309f9f0ccad-maas-api-tls" pod "c2f9d33f-085d-4744-85cc-d309f9f0ccad" (UID: "c2f9d33f-085d-4744-85cc-d309f9f0ccad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:40.242561 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:40.242542 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2f9d33f-085d-4744-85cc-d309f9f0ccad-metrics-certs" pod "c2f9d33f-085d-4744-85cc-d309f9f0ccad" (UID: "c2f9d33f-085d-4744-85cc-d309f9f0ccad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:40.242720 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:40.242694 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2f9d33f-085d-4744-85cc-d309f9f0ccad-kube-api-access-wm85w" pod "c2f9d33f-085d-4744-85cc-d309f9f0ccad" (UID: "c2f9d33f-085d-4744-85cc-d309f9f0ccad"). InnerVolumeSpecName "kube-api-access-wm85w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:40.343105 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:40.343066 4965 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2f9d33f-085d-4744-85cc-d309f9f0ccad-maas-api-tls\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:25:40.343105 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:40.343106 4965 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2f9d33f-085d-4744-85cc-d309f9f0ccad-metrics-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:25:40.343368 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:40.343122 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm85w\" (UniqueName: \"kubernetes.io/projected/c2f9d33f-085d-4744-85cc-d309f9f0ccad-kube-api-access-wm85w\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:25:40.745126 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:40.745085 4965 generic.go:356] "Generic (PLEG): container finished" podID="c2f9d33f-085d-4744-85cc-d309f9f0ccad" containerID="12f071f9e0f895805ac2ee7dfc3225b7d06c08231a38be734410754a6c2eb9c5" exitCode=0 Oct 30 18:25:40.745356 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:40.745172 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-6tb6z" Oct 30 18:25:40.745356 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:40.745178 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-6tb6z" event={"ID":"c2f9d33f-085d-4744-85cc-d309f9f0ccad","Type":"ContainerDied","Data":"12f071f9e0f895805ac2ee7dfc3225b7d06c08231a38be734410754a6c2eb9c5"} Oct 30 18:25:40.745356 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:40.745278 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-6tb6z" event={"ID":"c2f9d33f-085d-4744-85cc-d309f9f0ccad","Type":"ContainerDied","Data":"93638b38513dd17d6016bb49f46472ba45c827c8b2cedd2c1d9a5d16be374743"} Oct 30 18:25:40.745356 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:40.745312 4965 scope.go:122] "RemoveContainer" containerID="12f071f9e0f895805ac2ee7dfc3225b7d06c08231a38be734410754a6c2eb9c5" Oct 30 18:25:40.758111 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:40.758088 4965 scope.go:122] "RemoveContainer" containerID="12f071f9e0f895805ac2ee7dfc3225b7d06c08231a38be734410754a6c2eb9c5" Oct 30 18:25:40.758466 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:25:40.758435 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12f071f9e0f895805ac2ee7dfc3225b7d06c08231a38be734410754a6c2eb9c5\": container with ID starting with 12f071f9e0f895805ac2ee7dfc3225b7d06c08231a38be734410754a6c2eb9c5 not found: ID does not exist" containerID="12f071f9e0f895805ac2ee7dfc3225b7d06c08231a38be734410754a6c2eb9c5" Oct 30 18:25:40.758559 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:40.758476 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12f071f9e0f895805ac2ee7dfc3225b7d06c08231a38be734410754a6c2eb9c5"} err="failed to get container status \"12f071f9e0f895805ac2ee7dfc3225b7d06c08231a38be734410754a6c2eb9c5\": rpc error: code = NotFound desc = could not find container \"12f071f9e0f895805ac2ee7dfc3225b7d06c08231a38be734410754a6c2eb9c5\": container with ID starting with 12f071f9e0f895805ac2ee7dfc3225b7d06c08231a38be734410754a6c2eb9c5 not found: ID does not exist" Oct 30 18:25:40.797893 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:40.797711 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-6tb6z"] Oct 30 18:25:40.809682 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:40.809645 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-6tb6z"] Oct 30 18:25:42.111151 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:42.111104 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2f9d33f-085d-4744-85cc-d309f9f0ccad" path="/var/lib/kubelet/pods/c2f9d33f-085d-4744-85cc-d309f9f0ccad/volumes" Oct 30 18:25:43.000924 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:43.000853 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-5c48877c75-6tb6z" podUID="c2f9d33f-085d-4744-85cc-d309f9f0ccad" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.44:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:47.825600 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:25:47.825561 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:26:01.148564 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:26:01.148524 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:26:06.047622 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:26:06.047583 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:26:34.848876 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:26:34.848772 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kqf65"] Oct 30 18:26:36.049314 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:26:36.049268 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:26:56.327921 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:26:56.327885 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-4b6zt_1f3cc1b1-0978-4c20-bea2-96329cc8d1c3/manager/0.log" Oct 30 18:26:57.935148 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:26:57.935115 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg_485b42c7-549a-479f-a4f9-62622fd84522/manager/0.log" Oct 30 18:26:57.966405 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:26:57.966380 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg_485b42c7-549a-479f-a4f9-62622fd84522/copy-manifests/0.log" Oct 30 18:27:05.995325 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:27:05.995288 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:27:19.509405 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:27:19.509373 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-9d288_a4c0219a-9aa4-434b-ab8d-b6385fbf58bc/maas-api/0.log" Oct 30 18:27:21.122852 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:27:21.122812 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-pd82z_2ae50ffd-59e5-43e8-8e4a-b8080c9eddc0/postgres/0.log" Oct 30 18:27:22.947154 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:27:22.947122 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-m275c_62ded67d-b19f-4857-8852-f6b1f02c0323/manager/0.log" Oct 30 18:27:23.017192 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:27:23.017167 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 18:27:23.018641 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:27:23.018618 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 18:27:23.820395 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:27:23.820363 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-kqf65_31a77224-5836-451e-a601-0229e81d4e74/limitador/0.log" Oct 30 18:27:25.019630 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:27:25.019597 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-kpfjj_284ed8f0-77ec-4f6c-8dd1-326a600ae840/discovery/0.log" Oct 30 18:27:25.395014 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:27:25.394976 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57db9d4f85-w5gx5_463b3376-b504-4399-be15-f81c8e1952e3/kube-auth-proxy/0.log" Oct 30 18:27:35.995786 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:27:35.995746 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:27:37.020264 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:27:37.020230 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-ms6c5_f835cdef-6be0-4f7d-b4c1-6e5a771e8616/istio-proxy/0.log" Oct 30 18:27:37.230397 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:27:37.230368 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-pvlkq_0c1403e9-4637-49cc-9e50-f7be9caa1847/istio-proxy/0.log" Oct 30 18:27:37.433381 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:27:37.433348 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-rkm25_9e0d78c2-d701-4ecc-888c-d69469f844ee/istio-proxy/0.log" Oct 30 18:27:37.636227 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:27:37.636177 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-zhw6k_ba865430-c09d-4c91-99f4-183ab4828a2a/payload-pre-processing/0.log" Oct 30 18:27:37.821856 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:27:37.821767 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-5tdrf_d383c0b3-9245-4397-b031-38295a768fba/payload-processing/0.log" Oct 30 18:27:38.032191 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:27:38.032161 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-ddb54c4f7-54tw2_924ac066-ff98-4320-9eb2-268f0a39ca33/router/0.log" Oct 30 18:27:39.292476 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:27:39.292442 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-q6l6h_c6416895-72e8-4440-9d81-f5854fed4aa5/main/0.log" Oct 30 18:27:39.503102 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:27:39.503072 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-st6hg_393bb669-2192-4775-b63c-f96a2aba0cd4/main/0.log" Oct 30 18:27:39.891150 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:27:39.891118 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs_d8d199b0-9f65-4d32-aafb-62e8dc216076/main/0.log" Oct 30 18:27:50.687071 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:27:50.687023 4965 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-222.ec2.internal" podUID="67161c7401d4577fb0911752be0668c2" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:05.996517 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:28:05.996421 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:28:30.696883 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:28:30.696840 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-4b6zt_1f3cc1b1-0978-4c20-bea2-96329cc8d1c3/manager/0.log" Oct 30 18:28:35.995120 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:28:35.995078 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:28:42.351032 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:28:42.350994 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg_485b42c7-549a-479f-a4f9-62622fd84522/copy-manifests/0.log" Oct 30 18:28:42.384899 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:28:42.384844 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c8cd7c7d9-9m4gg_485b42c7-549a-479f-a4f9-62622fd84522/manager/0.log" Oct 30 18:28:43.633041 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:28:43.633012 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-9d288_a4c0219a-9aa4-434b-ab8d-b6385fbf58bc/maas-api/0.log" Oct 30 18:29:05.003533 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:29:05.003498 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-pd82z_2ae50ffd-59e5-43e8-8e4a-b8080c9eddc0/postgres/0.log" Oct 30 18:29:05.995606 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:29:05.995572 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:29:06.426336 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:29:06.426299 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-m275c_62ded67d-b19f-4857-8852-f6b1f02c0323/manager/0.log" Oct 30 18:29:07.281328 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:29:07.281300 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-kqf65_31a77224-5836-451e-a601-0229e81d4e74/limitador/0.log" Oct 30 18:29:18.324395 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:29:18.324362 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-kpfjj_284ed8f0-77ec-4f6c-8dd1-326a600ae840/discovery/0.log" Oct 30 18:29:18.754988 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:29:18.754958 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57db9d4f85-w5gx5_463b3376-b504-4399-be15-f81c8e1952e3/kube-auth-proxy/0.log" Oct 30 18:29:20.400320 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:29:20.400289 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-ms6c5_f835cdef-6be0-4f7d-b4c1-6e5a771e8616/istio-proxy/0.log" Oct 30 18:29:20.600024 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:29:20.599993 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-pvlkq_0c1403e9-4637-49cc-9e50-f7be9caa1847/istio-proxy/0.log" Oct 30 18:29:20.797952 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:29:20.797874 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-rkm25_9e0d78c2-d701-4ecc-888c-d69469f844ee/istio-proxy/0.log" Oct 30 18:29:20.977808 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:29:20.977773 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-zhw6k_ba865430-c09d-4c91-99f4-183ab4828a2a/payload-pre-processing/0.log" Oct 30 18:29:21.160573 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:29:21.160545 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-5tdrf_d383c0b3-9245-4397-b031-38295a768fba/payload-processing/0.log" Oct 30 18:29:21.360791 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:29:21.360757 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-ddb54c4f7-54tw2_924ac066-ff98-4320-9eb2-268f0a39ca33/router/0.log" Oct 30 18:29:32.291484 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:29:32.291454 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-q6l6h_c6416895-72e8-4440-9d81-f5854fed4aa5/main/0.log" Oct 30 18:29:35.996202 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:29:35.996110 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:29:42.496699 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:29:42.496667 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-st6hg_393bb669-2192-4775-b63c-f96a2aba0cd4/main/0.log" Oct 30 18:29:42.892489 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:29:42.892460 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-dl9vs_d8d199b0-9f65-4d32-aafb-62e8dc216076/main/0.log" Oct 30 18:30:05.996322 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:30:05.996289 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:30:35.994932 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:30:35.994897 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:31:05.995998 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:31:05.995914 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:31:27.749458 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:27.749428 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-5tdrf_d383c0b3-9245-4397-b031-38295a768fba/payload-processing/0.log" Oct 30 18:31:27.943831 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:27.943802 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-5tdrf_d383c0b3-9245-4397-b031-38295a768fba/payload-processing/0.log" Oct 30 18:31:35.998575 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:31:35.998535 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:31:51.832101 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:51.832061 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25"] Oct 30 18:31:51.833095 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:51.832323 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" podUID="9e0d78c2-d701-4ecc-888c-d69469f844ee" containerName="istio-proxy" containerID="cri-o://7b5c84018529e842a8e1b51e172f6627918577b548c8171bd207f7acd16f7f1c" gracePeriod=30 Oct 30 18:31:57.084943 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.084916 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:31:57.191590 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.191514 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e0d78c2-d701-4ecc-888c-d69469f844ee-kube-api-access-5tx5q\" (UniqueName: \"kubernetes.io/projected/9e0d78c2-d701-4ecc-888c-d69469f844ee-kube-api-access-5tx5q\") pod \"9e0d78c2-d701-4ecc-888c-d69469f844ee\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " Oct 30 18:31:57.191770 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.191586 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-istio-envoy\") pod \"9e0d78c2-d701-4ecc-888c-d69469f844ee\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " Oct 30 18:31:57.191770 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.191668 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-istio-data\") pod \"9e0d78c2-d701-4ecc-888c-d69469f844ee\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " Oct 30 18:31:57.191770 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.191718 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-workload-certs\") pod \"9e0d78c2-d701-4ecc-888c-d69469f844ee\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " Oct 30 18:31:57.191971 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.191785 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e0d78c2-d701-4ecc-888c-d69469f844ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e0d78c2-d701-4ecc-888c-d69469f844ee-istiod-ca-cert\") pod \"9e0d78c2-d701-4ecc-888c-d69469f844ee\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " Oct 30 18:31:57.191971 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.191821 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-credential-socket\") pod \"9e0d78c2-d701-4ecc-888c-d69469f844ee\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " Oct 30 18:31:57.191971 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.191856 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e0d78c2-d701-4ecc-888c-d69469f844ee-istio-token\" (UniqueName: \"kubernetes.io/projected/9e0d78c2-d701-4ecc-888c-d69469f844ee-istio-token\") pod \"9e0d78c2-d701-4ecc-888c-d69469f844ee\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " Oct 30 18:31:57.191971 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.191903 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e0d78c2-d701-4ecc-888c-d69469f844ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e0d78c2-d701-4ecc-888c-d69469f844ee-istio-podinfo\") pod \"9e0d78c2-d701-4ecc-888c-d69469f844ee\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " Oct 30 18:31:57.191971 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.191942 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-workload-socket\") pod \"9e0d78c2-d701-4ecc-888c-d69469f844ee\" (UID: \"9e0d78c2-d701-4ecc-888c-d69469f844ee\") " Oct 30 18:31:57.191971 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.191933 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-istio-data" pod "9e0d78c2-d701-4ecc-888c-d69469f844ee" (UID: "9e0d78c2-d701-4ecc-888c-d69469f844ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:31:57.192288 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.192018 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-workload-certs" pod "9e0d78c2-d701-4ecc-888c-d69469f844ee" (UID: "9e0d78c2-d701-4ecc-888c-d69469f844ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:31:57.192288 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.192247 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-workload-socket" pod "9e0d78c2-d701-4ecc-888c-d69469f844ee" (UID: "9e0d78c2-d701-4ecc-888c-d69469f844ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:31:57.192396 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.192367 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-credential-socket" pod "9e0d78c2-d701-4ecc-888c-d69469f844ee" (UID: "9e0d78c2-d701-4ecc-888c-d69469f844ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:31:57.192396 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.192247 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e0d78c2-d701-4ecc-888c-d69469f844ee-istiod-ca-cert" pod "9e0d78c2-d701-4ecc-888c-d69469f844ee" (UID: "9e0d78c2-d701-4ecc-888c-d69469f844ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:31:57.192945 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.192569 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:31:57.192945 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.192728 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:31:57.192945 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.192753 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:31:57.194886 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.194311 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e0d78c2-d701-4ecc-888c-d69469f844ee-istio-podinfo" pod "9e0d78c2-d701-4ecc-888c-d69469f844ee" (UID: "9e0d78c2-d701-4ecc-888c-d69469f844ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:31:57.194886 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.194691 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e0d78c2-d701-4ecc-888c-d69469f844ee-kube-api-access-5tx5q" pod "9e0d78c2-d701-4ecc-888c-d69469f844ee" (UID: "9e0d78c2-d701-4ecc-888c-d69469f844ee"). InnerVolumeSpecName "kube-api-access-5tx5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:31:57.195099 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.194899 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e0d78c2-d701-4ecc-888c-d69469f844ee-istio-token" pod "9e0d78c2-d701-4ecc-888c-d69469f844ee" (UID: "9e0d78c2-d701-4ecc-888c-d69469f844ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:31:57.195260 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.195237 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-istio-envoy" pod "9e0d78c2-d701-4ecc-888c-d69469f844ee" (UID: "9e0d78c2-d701-4ecc-888c-d69469f844ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:31:57.293798 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.293764 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e0d78c2-d701-4ecc-888c-d69469f844ee-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:31:57.293798 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.293790 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:31:57.293798 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.293800 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e0d78c2-d701-4ecc-888c-d69469f844ee-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:31:57.294051 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.293810 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e0d78c2-d701-4ecc-888c-d69469f844ee-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:31:57.294051 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.293819 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tx5q\" (UniqueName: \"kubernetes.io/projected/9e0d78c2-d701-4ecc-888c-d69469f844ee-kube-api-access-5tx5q\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:31:57.294051 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.293830 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e0d78c2-d701-4ecc-888c-d69469f844ee-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:31:57.625533 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.625492 4965 generic.go:356] "Generic (PLEG): container finished" podID="9e0d78c2-d701-4ecc-888c-d69469f844ee" containerID="7b5c84018529e842a8e1b51e172f6627918577b548c8171bd207f7acd16f7f1c" exitCode=0 Oct 30 18:31:57.625716 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.625546 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" Oct 30 18:31:57.625716 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.625582 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" event={"ID":"9e0d78c2-d701-4ecc-888c-d69469f844ee","Type":"ContainerDied","Data":"7b5c84018529e842a8e1b51e172f6627918577b548c8171bd207f7acd16f7f1c"} Oct 30 18:31:57.625716 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.625638 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25" event={"ID":"9e0d78c2-d701-4ecc-888c-d69469f844ee","Type":"ContainerDied","Data":"91e73e3a005bbf4f4de99c841f0354ed55dc03b6e0bd9fa274c17b64d161cb84"} Oct 30 18:31:57.625716 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.625659 4965 scope.go:122] "RemoveContainer" containerID="7b5c84018529e842a8e1b51e172f6627918577b548c8171bd207f7acd16f7f1c" Oct 30 18:31:57.635849 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.635825 4965 scope.go:122] "RemoveContainer" containerID="7b5c84018529e842a8e1b51e172f6627918577b548c8171bd207f7acd16f7f1c" Oct 30 18:31:57.636208 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:31:57.636186 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b5c84018529e842a8e1b51e172f6627918577b548c8171bd207f7acd16f7f1c\": container with ID starting with 7b5c84018529e842a8e1b51e172f6627918577b548c8171bd207f7acd16f7f1c not found: ID does not exist" containerID="7b5c84018529e842a8e1b51e172f6627918577b548c8171bd207f7acd16f7f1c" Oct 30 18:31:57.636293 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.636214 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b5c84018529e842a8e1b51e172f6627918577b548c8171bd207f7acd16f7f1c"} err="failed to get container status \"7b5c84018529e842a8e1b51e172f6627918577b548c8171bd207f7acd16f7f1c\": rpc error: code = NotFound desc = could not find container \"7b5c84018529e842a8e1b51e172f6627918577b548c8171bd207f7acd16f7f1c\": container with ID starting with 7b5c84018529e842a8e1b51e172f6627918577b548c8171bd207f7acd16f7f1c not found: ID does not exist" Oct 30 18:31:57.689880 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.689839 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25"] Oct 30 18:31:57.701950 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:57.701918 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rkm25"] Oct 30 18:31:58.097157 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:31:58.097127 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e0d78c2-d701-4ecc-888c-d69469f844ee" path="/var/lib/kubelet/pods/9e0d78c2-d701-4ecc-888c-d69469f844ee/volumes" Oct 30 18:32:05.996697 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:32:05.996591 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:32:06.774350 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:06.774310 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq"] Oct 30 18:32:06.774636 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:06.774612 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" podUID="0c1403e9-4637-49cc-9e50-f7be9caa1847" containerName="istio-proxy" containerID="cri-o://565c35901d80889fffe733d06e2114cd7ea4709590807a38a5d8c5e68d730c01" gracePeriod=30 Oct 30 18:32:06.800906 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:06.787447 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5"] Oct 30 18:32:06.800906 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:06.787697 4965 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" podUID="f835cdef-6be0-4f7d-b4c1-6e5a771e8616" containerName="istio-proxy" containerID="cri-o://a850a12f8d82f7f3b341312d2de62c9ae3b7cc87fc99a557b76837c8ad24cc1b" gracePeriod=30 Oct 30 18:32:08.660216 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:08.660181 4965 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ms6c5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:32:08.660602 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:08.660244 4965 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" podUID="f835cdef-6be0-4f7d-b4c1-6e5a771e8616" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:32:12.039761 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.039734 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:32:12.066535 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.066486 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 18:32:12.143489 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.143460 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-istio-data\") pod \"0c1403e9-4637-49cc-9e50-f7be9caa1847\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " Oct 30 18:32:12.143657 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.143549 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-credential-socket\") pod \"0c1403e9-4637-49cc-9e50-f7be9caa1847\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " Oct 30 18:32:12.143657 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.143571 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c1403e9-4637-49cc-9e50-f7be9caa1847-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c1403e9-4637-49cc-9e50-f7be9caa1847-istiod-ca-cert\") pod \"0c1403e9-4637-49cc-9e50-f7be9caa1847\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " Oct 30 18:32:12.143657 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.143597 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-workload-certs\") pod \"0c1403e9-4637-49cc-9e50-f7be9caa1847\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " Oct 30 18:32:12.143657 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.143628 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-workload-certs\") pod \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " Oct 30 18:32:12.143925 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.143657 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istio-data\") pod \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " Oct 30 18:32:12.143925 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.143690 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-kube-api-access-wkghc\" (UniqueName: \"kubernetes.io/projected/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-kube-api-access-wkghc\") pod \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " Oct 30 18:32:12.143925 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.143721 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-istio-envoy\") pod \"0c1403e9-4637-49cc-9e50-f7be9caa1847\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " Oct 30 18:32:12.143925 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.143747 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c1403e9-4637-49cc-9e50-f7be9caa1847-istio-token\" (UniqueName: \"kubernetes.io/projected/0c1403e9-4637-49cc-9e50-f7be9caa1847-istio-token\") pod \"0c1403e9-4637-49cc-9e50-f7be9caa1847\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " Oct 30 18:32:12.143925 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.143801 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istio-token\" (UniqueName: \"kubernetes.io/projected/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istio-token\") pod \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " Oct 30 18:32:12.143925 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.143806 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-istio-data" pod "0c1403e9-4637-49cc-9e50-f7be9caa1847" (UID: "0c1403e9-4637-49cc-9e50-f7be9caa1847"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:12.143925 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.143829 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-workload-certs" pod "0c1403e9-4637-49cc-9e50-f7be9caa1847" (UID: "0c1403e9-4637-49cc-9e50-f7be9caa1847"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:12.143925 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.143832 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-credential-socket\") pod \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " Oct 30 18:32:12.143925 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.143890 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-workload-certs" pod "f835cdef-6be0-4f7d-b4c1-6e5a771e8616" (UID: "f835cdef-6be0-4f7d-b4c1-6e5a771e8616"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:12.143925 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.143917 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istiod-ca-cert\") pod \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " Oct 30 18:32:12.144497 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.143941 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-credential-socket" pod "0c1403e9-4637-49cc-9e50-f7be9caa1847" (UID: "0c1403e9-4637-49cc-9e50-f7be9caa1847"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:12.144497 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.143954 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istio-envoy\") pod \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " Oct 30 18:32:12.144497 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.143992 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c1403e9-4637-49cc-9e50-f7be9caa1847-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c1403e9-4637-49cc-9e50-f7be9caa1847-istio-podinfo\") pod \"0c1403e9-4637-49cc-9e50-f7be9caa1847\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " Oct 30 18:32:12.144497 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.144043 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istio-podinfo\") pod \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " Oct 30 18:32:12.144497 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.144083 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-workload-socket\") pod \"0c1403e9-4637-49cc-9e50-f7be9caa1847\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " Oct 30 18:32:12.144497 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.144112 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-workload-socket\") pod \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\" (UID: \"f835cdef-6be0-4f7d-b4c1-6e5a771e8616\") " Oct 30 18:32:12.144497 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.144132 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-credential-socket" pod "f835cdef-6be0-4f7d-b4c1-6e5a771e8616" (UID: "f835cdef-6be0-4f7d-b4c1-6e5a771e8616"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:12.144497 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.144145 4965 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c1403e9-4637-49cc-9e50-f7be9caa1847-kube-api-access-j2zr6\" (UniqueName: \"kubernetes.io/projected/0c1403e9-4637-49cc-9e50-f7be9caa1847-kube-api-access-j2zr6\") pod \"0c1403e9-4637-49cc-9e50-f7be9caa1847\" (UID: \"0c1403e9-4637-49cc-9e50-f7be9caa1847\") " Oct 30 18:32:12.144497 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.144274 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istio-data" pod "f835cdef-6be0-4f7d-b4c1-6e5a771e8616" (UID: "f835cdef-6be0-4f7d-b4c1-6e5a771e8616"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:12.144497 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.144304 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istiod-ca-cert" pod "f835cdef-6be0-4f7d-b4c1-6e5a771e8616" (UID: "f835cdef-6be0-4f7d-b4c1-6e5a771e8616"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:32:12.144497 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.144306 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c1403e9-4637-49cc-9e50-f7be9caa1847-istiod-ca-cert" pod "0c1403e9-4637-49cc-9e50-f7be9caa1847" (UID: "0c1403e9-4637-49cc-9e50-f7be9caa1847"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:32:12.145141 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.144557 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:32:12.145141 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.144574 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-workload-socket" pod "f835cdef-6be0-4f7d-b4c1-6e5a771e8616" (UID: "f835cdef-6be0-4f7d-b4c1-6e5a771e8616"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:12.145141 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.144581 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:32:12.145141 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.144614 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:32:12.145141 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.144629 4965 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-credential-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:32:12.145141 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.144643 4965 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c1403e9-4637-49cc-9e50-f7be9caa1847-istiod-ca-cert\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:32:12.145141 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.144657 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:32:12.145141 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.144671 4965 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-workload-certs\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:32:12.145141 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.144683 4965 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istio-data\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:32:12.145687 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.145225 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-workload-socket" pod "0c1403e9-4637-49cc-9e50-f7be9caa1847" (UID: "0c1403e9-4637-49cc-9e50-f7be9caa1847"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:12.146945 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.146723 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-istio-envoy" pod "0c1403e9-4637-49cc-9e50-f7be9caa1847" (UID: "0c1403e9-4637-49cc-9e50-f7be9caa1847"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:12.146945 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.146885 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istio-envoy" pod "f835cdef-6be0-4f7d-b4c1-6e5a771e8616" (UID: "f835cdef-6be0-4f7d-b4c1-6e5a771e8616"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:12.146945 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.146904 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istio-podinfo" pod "f835cdef-6be0-4f7d-b4c1-6e5a771e8616" (UID: "f835cdef-6be0-4f7d-b4c1-6e5a771e8616"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:32:12.146945 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.146922 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istio-token" pod "f835cdef-6be0-4f7d-b4c1-6e5a771e8616" (UID: "f835cdef-6be0-4f7d-b4c1-6e5a771e8616"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:12.147261 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.147018 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-kube-api-access-wkghc" pod "f835cdef-6be0-4f7d-b4c1-6e5a771e8616" (UID: "f835cdef-6be0-4f7d-b4c1-6e5a771e8616"). InnerVolumeSpecName "kube-api-access-wkghc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:12.147333 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.147264 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c1403e9-4637-49cc-9e50-f7be9caa1847-kube-api-access-j2zr6" pod "0c1403e9-4637-49cc-9e50-f7be9caa1847" (UID: "0c1403e9-4637-49cc-9e50-f7be9caa1847"). InnerVolumeSpecName "kube-api-access-j2zr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:12.147418 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.147399 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c1403e9-4637-49cc-9e50-f7be9caa1847-istio-token" pod "0c1403e9-4637-49cc-9e50-f7be9caa1847" (UID: "0c1403e9-4637-49cc-9e50-f7be9caa1847"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:12.148475 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.148451 4965 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c1403e9-4637-49cc-9e50-f7be9caa1847-istio-podinfo" pod "0c1403e9-4637-49cc-9e50-f7be9caa1847" (UID: "0c1403e9-4637-49cc-9e50-f7be9caa1847"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:32:12.245982 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.245946 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:32:12.245982 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.245978 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c1403e9-4637-49cc-9e50-f7be9caa1847-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:32:12.245982 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.245986 4965 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istio-token\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:32:12.246228 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.245994 4965 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istio-envoy\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:32:12.246228 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.246002 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c1403e9-4637-49cc-9e50-f7be9caa1847-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:32:12.246228 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.246009 4965 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-istio-podinfo\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:32:12.246228 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.246018 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1403e9-4637-49cc-9e50-f7be9caa1847-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:32:12.246228 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.246027 4965 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-workload-socket\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:32:12.246228 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.246035 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2zr6\" (UniqueName: \"kubernetes.io/projected/0c1403e9-4637-49cc-9e50-f7be9caa1847-kube-api-access-j2zr6\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:32:12.246228 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.246044 4965 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkghc\" (UniqueName: \"kubernetes.io/projected/f835cdef-6be0-4f7d-b4c1-6e5a771e8616-kube-api-access-wkghc\") on node \"ip-10-0-139-222.ec2.internal\" DevicePath \"\"" Oct 30 18:32:12.698944 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.698912 4965 generic.go:356] "Generic (PLEG): container finished" podID="f835cdef-6be0-4f7d-b4c1-6e5a771e8616" containerID="a850a12f8d82f7f3b341312d2de62c9ae3b7cc87fc99a557b76837c8ad24cc1b" exitCode=0 Oct 30 18:32:12.699153 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.698963 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" Oct 30 18:32:12.699153 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.698999 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" event={"ID":"f835cdef-6be0-4f7d-b4c1-6e5a771e8616","Type":"ContainerDied","Data":"a850a12f8d82f7f3b341312d2de62c9ae3b7cc87fc99a557b76837c8ad24cc1b"} Oct 30 18:32:12.699153 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.699050 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5" event={"ID":"f835cdef-6be0-4f7d-b4c1-6e5a771e8616","Type":"ContainerDied","Data":"70508f315a352918865d25f49f2a68f2b4b564725ca426efd15a0bb51c33cb5b"} Oct 30 18:32:12.699153 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.699076 4965 scope.go:122] "RemoveContainer" containerID="a850a12f8d82f7f3b341312d2de62c9ae3b7cc87fc99a557b76837c8ad24cc1b" Oct 30 18:32:12.700383 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.700361 4965 generic.go:356] "Generic (PLEG): container finished" podID="0c1403e9-4637-49cc-9e50-f7be9caa1847" containerID="565c35901d80889fffe733d06e2114cd7ea4709590807a38a5d8c5e68d730c01" exitCode=0 Oct 30 18:32:12.700477 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.700404 4965 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" Oct 30 18:32:12.700477 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.700448 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" event={"ID":"0c1403e9-4637-49cc-9e50-f7be9caa1847","Type":"ContainerDied","Data":"565c35901d80889fffe733d06e2114cd7ea4709590807a38a5d8c5e68d730c01"} Oct 30 18:32:12.700610 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.700498 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq" event={"ID":"0c1403e9-4637-49cc-9e50-f7be9caa1847","Type":"ContainerDied","Data":"85fa8b4017d7a2f9e4771340ec641c7f251780d117e9b0bb3af643022efa9269"} Oct 30 18:32:12.708934 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.708911 4965 scope.go:122] "RemoveContainer" containerID="a850a12f8d82f7f3b341312d2de62c9ae3b7cc87fc99a557b76837c8ad24cc1b" Oct 30 18:32:12.709265 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:32:12.709225 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a850a12f8d82f7f3b341312d2de62c9ae3b7cc87fc99a557b76837c8ad24cc1b\": container with ID starting with a850a12f8d82f7f3b341312d2de62c9ae3b7cc87fc99a557b76837c8ad24cc1b not found: ID does not exist" containerID="a850a12f8d82f7f3b341312d2de62c9ae3b7cc87fc99a557b76837c8ad24cc1b" Oct 30 18:32:12.709265 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.709250 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a850a12f8d82f7f3b341312d2de62c9ae3b7cc87fc99a557b76837c8ad24cc1b"} err="failed to get container status \"a850a12f8d82f7f3b341312d2de62c9ae3b7cc87fc99a557b76837c8ad24cc1b\": rpc error: code = NotFound desc = could not find container \"a850a12f8d82f7f3b341312d2de62c9ae3b7cc87fc99a557b76837c8ad24cc1b\": container with ID starting with a850a12f8d82f7f3b341312d2de62c9ae3b7cc87fc99a557b76837c8ad24cc1b not found: ID does not exist" Oct 30 18:32:12.709415 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.709272 4965 scope.go:122] "RemoveContainer" containerID="565c35901d80889fffe733d06e2114cd7ea4709590807a38a5d8c5e68d730c01" Oct 30 18:32:12.717266 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.717252 4965 scope.go:122] "RemoveContainer" containerID="565c35901d80889fffe733d06e2114cd7ea4709590807a38a5d8c5e68d730c01" Oct 30 18:32:12.717505 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:32:12.717479 4965 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"565c35901d80889fffe733d06e2114cd7ea4709590807a38a5d8c5e68d730c01\": container with ID starting with 565c35901d80889fffe733d06e2114cd7ea4709590807a38a5d8c5e68d730c01 not found: ID does not exist" containerID="565c35901d80889fffe733d06e2114cd7ea4709590807a38a5d8c5e68d730c01" Oct 30 18:32:12.717604 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.717506 4965 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"565c35901d80889fffe733d06e2114cd7ea4709590807a38a5d8c5e68d730c01"} err="failed to get container status \"565c35901d80889fffe733d06e2114cd7ea4709590807a38a5d8c5e68d730c01\": rpc error: code = NotFound desc = could not find container \"565c35901d80889fffe733d06e2114cd7ea4709590807a38a5d8c5e68d730c01\": container with ID starting with 565c35901d80889fffe733d06e2114cd7ea4709590807a38a5d8c5e68d730c01 not found: ID does not exist" Oct 30 18:32:12.751851 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.751826 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq"] Oct 30 18:32:12.764154 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.764130 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pvlkq"] Oct 30 18:32:12.790848 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.790822 4965 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5"] Oct 30 18:32:12.797079 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:12.797057 4965 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ms6c5"] Oct 30 18:32:14.096267 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:14.096236 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c1403e9-4637-49cc-9e50-f7be9caa1847" path="/var/lib/kubelet/pods/0c1403e9-4637-49cc-9e50-f7be9caa1847/volumes" Oct 30 18:32:14.096688 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:14.096673 4965 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f835cdef-6be0-4f7d-b4c1-6e5a771e8616" path="/var/lib/kubelet/pods/f835cdef-6be0-4f7d-b4c1-6e5a771e8616/volumes" Oct 30 18:32:23.051448 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:23.051416 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 18:32:23.055373 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:23.055346 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 18:32:35.998836 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:32:35.998748 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:32:50.261800 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:50.261770 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-5tdrf_d383c0b3-9245-4397-b031-38295a768fba/payload-processing/0.log" Oct 30 18:32:50.452577 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:50.452549 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-5tdrf_d383c0b3-9245-4397-b031-38295a768fba/payload-processing/0.log" Oct 30 18:32:55.097793 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:55.097757 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5mtjr_2906e95e-3baa-4c79-ad9c-b2592c1d414f/global-pull-secret-syncer/0.log" Oct 30 18:32:55.739731 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:55.739702 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-9jx9m_c7ac81c7-e1f0-4549-886d-d7b8b893b614/connection-checker/0.log" Oct 30 18:32:55.871437 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:55.871411 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-hwfbh_e30e6b59-b8e2-4125-ae7a-41edd1ce7389/connection-checker/0.log" Oct 30 18:32:55.996885 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:55.996783 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-w5rnm_03a0e7f9-9e86-4aa9-9864-205263f6a486/connection-checker/0.log" Oct 30 18:32:56.318041 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:56.317968 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jnfks_f442e60f-8ccf-485e-9709-60a909d482bc/konnectivity-agent/0.log" Oct 30 18:32:56.733720 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:32:56.733685 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-222.ec2.internal_67161c7401d4577fb0911752be0668c2/haproxy/0.log" Oct 30 18:33:03.234320 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:03.234283 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-m275c_62ded67d-b19f-4857-8852-f6b1f02c0323/manager/0.log" Oct 30 18:33:03.856813 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:03.856778 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-kqf65_31a77224-5836-451e-a601-0229e81d4e74/limitador/0.log" Oct 30 18:33:04.696054 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.696018 4965 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cjt4j/perf-node-gather-daemonset-4gwxh"] Oct 30 18:33:04.696500 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.696420 4965 cpu_manager.go:414] "containerMap: removing container" podUID="f835cdef-6be0-4f7d-b4c1-6e5a771e8616" containerName="istio-proxy" Oct 30 18:33:04.696500 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.696433 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f835cdef-6be0-4f7d-b4c1-6e5a771e8616" containerName="istio-proxy" Oct 30 18:33:04.696500 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.696458 4965 cpu_manager.go:414] "containerMap: removing container" podUID="9e0d78c2-d701-4ecc-888c-d69469f844ee" containerName="istio-proxy" Oct 30 18:33:04.696500 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.696465 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e0d78c2-d701-4ecc-888c-d69469f844ee" containerName="istio-proxy" Oct 30 18:33:04.696500 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.696474 4965 cpu_manager.go:414] "containerMap: removing container" podUID="0c1403e9-4637-49cc-9e50-f7be9caa1847" containerName="istio-proxy" Oct 30 18:33:04.696500 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.696478 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c1403e9-4637-49cc-9e50-f7be9caa1847" containerName="istio-proxy" Oct 30 18:33:04.696500 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.696497 4965 cpu_manager.go:414] "containerMap: removing container" podUID="c2f9d33f-085d-4744-85cc-d309f9f0ccad" containerName="maas-api" Oct 30 18:33:04.696500 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.696501 4965 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2f9d33f-085d-4744-85cc-d309f9f0ccad" containerName="maas-api" Oct 30 18:33:04.696806 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.696574 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e0d78c2-d701-4ecc-888c-d69469f844ee" containerName="istio-proxy" Oct 30 18:33:04.696806 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.696583 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c1403e9-4637-49cc-9e50-f7be9caa1847" containerName="istio-proxy" Oct 30 18:33:04.696806 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.696592 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2f9d33f-085d-4744-85cc-d309f9f0ccad" containerName="maas-api" Oct 30 18:33:04.696806 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.696599 4965 memory_manager.go:365] "RemoveStaleState removing state" podUID="f835cdef-6be0-4f7d-b4c1-6e5a771e8616" containerName="istio-proxy" Oct 30 18:33:04.699620 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.699603 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-4gwxh" Oct 30 18:33:04.715481 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.715459 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cjt4j\"/\"kube-root-ca.crt\"" Oct 30 18:33:04.717580 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.717555 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cjt4j/perf-node-gather-daemonset-4gwxh"] Oct 30 18:33:04.797779 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.797749 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/453c53ea-5908-4ac6-943a-97847dc71407-lib-modules\") pod \"perf-node-gather-daemonset-4gwxh\" (UID: \"453c53ea-5908-4ac6-943a-97847dc71407\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-4gwxh" Oct 30 18:33:04.797779 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.797783 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/453c53ea-5908-4ac6-943a-97847dc71407-proc\") pod \"perf-node-gather-daemonset-4gwxh\" (UID: \"453c53ea-5908-4ac6-943a-97847dc71407\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-4gwxh" Oct 30 18:33:04.798048 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.797894 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/453c53ea-5908-4ac6-943a-97847dc71407-podres\") pod \"perf-node-gather-daemonset-4gwxh\" (UID: \"453c53ea-5908-4ac6-943a-97847dc71407\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-4gwxh" Oct 30 18:33:04.798048 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.797955 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/453c53ea-5908-4ac6-943a-97847dc71407-sys\") pod \"perf-node-gather-daemonset-4gwxh\" (UID: \"453c53ea-5908-4ac6-943a-97847dc71407\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-4gwxh" Oct 30 18:33:04.798048 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.797982 4965 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr8j6\" (UniqueName: \"kubernetes.io/projected/453c53ea-5908-4ac6-943a-97847dc71407-kube-api-access-zr8j6\") pod \"perf-node-gather-daemonset-4gwxh\" (UID: \"453c53ea-5908-4ac6-943a-97847dc71407\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-4gwxh" Oct 30 18:33:04.798741 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.798726 4965 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cjt4j\"/\"default-dockercfg-pcvs7\"" Oct 30 18:33:04.816062 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.816034 4965 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cjt4j\"/\"openshift-service-ca.crt\"" Oct 30 18:33:04.898829 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.898801 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/453c53ea-5908-4ac6-943a-97847dc71407-lib-modules\") pod \"perf-node-gather-daemonset-4gwxh\" (UID: \"453c53ea-5908-4ac6-943a-97847dc71407\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-4gwxh" Oct 30 18:33:04.899037 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.898835 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/453c53ea-5908-4ac6-943a-97847dc71407-proc\") pod \"perf-node-gather-daemonset-4gwxh\" (UID: \"453c53ea-5908-4ac6-943a-97847dc71407\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-4gwxh" Oct 30 18:33:04.899037 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.898895 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/453c53ea-5908-4ac6-943a-97847dc71407-podres\") pod \"perf-node-gather-daemonset-4gwxh\" (UID: \"453c53ea-5908-4ac6-943a-97847dc71407\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-4gwxh" Oct 30 18:33:04.899037 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.898915 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/453c53ea-5908-4ac6-943a-97847dc71407-sys\") pod \"perf-node-gather-daemonset-4gwxh\" (UID: \"453c53ea-5908-4ac6-943a-97847dc71407\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-4gwxh" Oct 30 18:33:04.899037 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.898932 4965 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr8j6\" (UniqueName: \"kubernetes.io/projected/453c53ea-5908-4ac6-943a-97847dc71407-kube-api-access-zr8j6\") pod \"perf-node-gather-daemonset-4gwxh\" (UID: \"453c53ea-5908-4ac6-943a-97847dc71407\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-4gwxh" Oct 30 18:33:04.899037 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.898970 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/453c53ea-5908-4ac6-943a-97847dc71407-lib-modules\") pod \"perf-node-gather-daemonset-4gwxh\" (UID: \"453c53ea-5908-4ac6-943a-97847dc71407\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-4gwxh" Oct 30 18:33:04.899037 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.899013 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/453c53ea-5908-4ac6-943a-97847dc71407-podres\") pod \"perf-node-gather-daemonset-4gwxh\" (UID: \"453c53ea-5908-4ac6-943a-97847dc71407\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-4gwxh" Oct 30 18:33:04.899037 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.899027 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/453c53ea-5908-4ac6-943a-97847dc71407-sys\") pod \"perf-node-gather-daemonset-4gwxh\" (UID: \"453c53ea-5908-4ac6-943a-97847dc71407\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-4gwxh" Oct 30 18:33:04.899301 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.899018 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/453c53ea-5908-4ac6-943a-97847dc71407-proc\") pod \"perf-node-gather-daemonset-4gwxh\" (UID: \"453c53ea-5908-4ac6-943a-97847dc71407\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-4gwxh" Oct 30 18:33:04.922492 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:04.922473 4965 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr8j6\" (UniqueName: \"kubernetes.io/projected/453c53ea-5908-4ac6-943a-97847dc71407-kube-api-access-zr8j6\") pod \"perf-node-gather-daemonset-4gwxh\" (UID: \"453c53ea-5908-4ac6-943a-97847dc71407\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-4gwxh" Oct 30 18:33:05.012477 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:05.012390 4965 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-4gwxh" Oct 30 18:33:05.316591 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:05.316562 4965 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cjt4j/perf-node-gather-daemonset-4gwxh"] Oct 30 18:33:05.317507 ip-10-0-139-222 kubenswrapper[4965]: W1030 18:33:05.317475 4965 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod453c53ea_5908_4ac6_943a_97847dc71407.slice/crio-ee98c07513b1018a25e584e7509e6aaf5165bd5c1463604c3d04f5420e90d144 WatchSource:0}: Error finding container ee98c07513b1018a25e584e7509e6aaf5165bd5c1463604c3d04f5420e90d144: Status 404 returned error can't find the container with id ee98c07513b1018a25e584e7509e6aaf5165bd5c1463604c3d04f5420e90d144 Oct 30 18:33:05.949763 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:05.949728 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-4gwxh" event={"ID":"453c53ea-5908-4ac6-943a-97847dc71407","Type":"ContainerStarted","Data":"8b744576b045a9ea7711851cc690b64e4e2e32593918b1b354f14730577911ad"} Oct 30 18:33:05.950230 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:05.949790 4965 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-4gwxh" event={"ID":"453c53ea-5908-4ac6-943a-97847dc71407","Type":"ContainerStarted","Data":"ee98c07513b1018a25e584e7509e6aaf5165bd5c1463604c3d04f5420e90d144"} Oct 30 18:33:05.950230 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:05.949890 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-4gwxh" Oct 30 18:33:05.990815 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:05.990750 4965 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-4gwxh" podStartSLOduration=1.990732268 podStartE2EDuration="1.990732268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:05.989582852 +0000 UTC m=+4844.041682783" watchObservedRunningTime="2026-10-30 18:33:05.990732268 +0000 UTC m=+4844.042832294" Oct 30 18:33:06.000750 ip-10-0-139-222 kubenswrapper[4965]: E1030 18:33:06.000720 4965 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tbqhc" containerName="extractor" Oct 30 18:33:06.396766 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:06.396736 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-6jbgg_cc71466d-eac0-460b-af50-3204f07b3f35/cluster-monitoring-operator/0.log" Oct 30 18:33:07.177901 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:07.177871 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bh2zb_066821ed-2f29-44e7-924d-00ac9d0d6273/node-exporter/0.log" Oct 30 18:33:07.295635 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:07.295607 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bh2zb_066821ed-2f29-44e7-924d-00ac9d0d6273/kube-rbac-proxy/0.log" Oct 30 18:33:07.414431 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:07.414408 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bh2zb_066821ed-2f29-44e7-924d-00ac9d0d6273/init-textfile/0.log" Oct 30 18:33:11.964322 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:11.964290 4965 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-4gwxh" Oct 30 18:33:12.497332 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:12.497300 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-kvpw6_c2d3293d-1890-4fa2-b790-9f84622c9c8b/networking-console-plugin/0.log" Oct 30 18:33:13.685647 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:13.685616 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/1.log" Oct 30 18:33:13.691367 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:13.691348 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mf9rd_1eab122e-07da-4307-b7f2-914cbe2b4237/console-operator/2.log" Oct 30 18:33:14.851395 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:14.851364 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-j6pg9_99c91f4b-d893-45d0-82a0-abbb240a49e7/download-server/0.log" Oct 30 18:33:16.023555 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:16.023521 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-4x66d_2989c0bb-0ed6-41b2-ad3e-bb6e85bd60b9/volume-data-source-validator/0.log" Oct 30 18:33:18.372275 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:18.372245 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-22vlj_2939b3a2-1950-44c9-802c-a53861aa7aa8/dns/0.log" Oct 30 18:33:18.488797 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:18.488767 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-22vlj_2939b3a2-1950-44c9-802c-a53861aa7aa8/kube-rbac-proxy/0.log" Oct 30 18:33:19.252428 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:19.252389 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zgb2b_467187ee-7039-4bb4-adbe-38161ee7c336/dns-node-resolver/0.log" Oct 30 18:33:19.738821 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:19.738782 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-kpfjj_284ed8f0-77ec-4f6c-8dd1-326a600ae840/discovery/0.log" Oct 30 18:33:19.841545 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:19.841511 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57db9d4f85-w5gx5_463b3376-b504-4399-be15-f81c8e1952e3/kube-auth-proxy/0.log" Oct 30 18:33:19.976161 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:19.976132 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-zhw6k_ba865430-c09d-4c91-99f4-183ab4828a2a/payload-pre-processing/0.log" Oct 30 18:33:20.040262 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:20.040186 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-5tdrf_d383c0b3-9245-4397-b031-38295a768fba/payload-processing/0.log" Oct 30 18:33:20.110352 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:20.110323 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-ddb54c4f7-54tw2_924ac066-ff98-4320-9eb2-268f0a39ca33/router/0.log" Oct 30 18:33:20.499689 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:20.499658 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-65f579c9f9-fhj5r_146cf698-06cf-47f6-a8fc-374eff2141be/registry/0.log" Oct 30 18:33:20.731832 ip-10-0-139-222 kubenswrapper[4965]: I1030 18:33:20.731802 4965 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-stqrx_2664a881-44d7-470a-93d0-df1b3eaca0bc/node-ca/0.log"